Modernizing a Legacy Java Application With OpenRewrite: What We Automated and What We Didn't
AI-Assisted Product Discovery
Distributed systems usually spread data by hashing keys across nodes so no one node carries all the load. A hot key breaks that assumption. It is a key whose access volume greatly exceeds the volume of other keys during a given period. A global view counter or a popular item identifier can receive millions of reads or writes each second. Every request targets the same partition, even when the rest of the cluster has spare capacity. The result is uneven load. One node reaches its CPU, network, thread, or connection limits while other nodes remain underused. Adding nodes does not solve the problem because traffic continues to resolve to the same partition. The overloaded shard becomes the bottleneck for a system that was intended to distribute work. Hot keys also create failure risks. If a popular cache entry expires or its host fails, many requests can fall through to the backing data store at once. This cache breakdown can overwhelm the database and trigger broader failures. Teams should monitor access patterns and alert when one key represents an unusual share of traffic, latency, or bandwidth. Split a Logical Key Into Physical Buckets A common mitigation is to split one logical key into several physical buckets. Instead of updating a single counter, a writer chooses a bucket and updates that bucket. The combined value is calculated by reading and aggregating all buckets. Java public void increment(String key) { int bucket = ThreadLocalRandom.current().nextInt(NUM_BUCKETS); jedis.incr(key + ":" + bucket); } public int readTotal(String key) { int total = 0; for (int i = 0; i < NUM_BUCKETS; i++) { total += Integer.parseInt(jedis.get(key + ":" + i)); } return total; } This pattern spreads writes across hash slots and nodes. The tradeoff is more work during reads. In Redis Cluster, a native `MGET`cannot read keys from different hash slots in one command. The client must issue individual reads, use parallel reads, or provide a scatter-gather layer that collects the bucket values. The aggregated total is also not an atomic snapshot because each bucket can be read at a different instant. Bucketed counters work well when approximate totals are acceptable, such as views, metrics, or telemetry. They are less appropriate when every read requires a strictly current, exact value. The number of buckets should reflect peak write volume, expected read frequency, and the cost of aggregation. Serve Read-Heavy Hot Keys From Local Memory When a hot key is primarily read, an application cache can reduce pressure on the shared store. A short-lived in-memory cache lets many requests reuse a recently fetched value before another request reaches Redis or a database. Java private static final LoadingCache<String, String> HOT_KEY_CACHE = Caffeine.newBuilder() .expireAfterWrite(50, TimeUnit.MILLISECONDS) .build(key -> { try (Jedis jedis = JEDIS_POOL.getResource()) { return jedis.get(key); } }); With this setup, most reads can be served from memory, and cache misses fetch the current value from Redis. The short expiration bounds the staleness window. This is useful when the business can tolerate a briefly outdated response in exchange for much lower read pressure on the shared data store. Systems can also use layered caches. A content delivery network or edge cache can absorb public traffic, followed by an application cache, then a distributed cache, and finally the durable store. Each layer should have a clear expiration policy and a plan for cache invalidation. The goal is to keep repeated reads from reaching the most constrained shared component. Replicate Values When Reads Are More Important Than Writes Another option for a read-dominated hot key is replication. The system stores several copies of the logical value on different shards, then distributes reads across the copies. Java int idx = ThreadLocalRandom.current().nextInt(REPLICA_COUNT); String replicaKey = originalKey + ":" + idx; String value = jedis.get(replicaKey); Replication can increase read capacity because different nodes serve independent requests. It adds work to writes because every replica must be updated. If an update fails for one replica, callers can see different values depending on the chosen key. This approach is useful when stale or temporarily inconsistent reads are acceptable. It needs an explicit repair strategy for failed replica updates. Smooth Write Spikes With Queues and Admission Control A hot key with a high write rate may need a different approach. Instead of applying every update immediately, the service can publish updates to a queue and let workers batch or aggregate them before writing to the shared store. This smooths spikes and protects the storage layer, but it adds write latency and creates a delay before the final value is visible. Rate limiting and backpressure can protect the system when a sudden surge would otherwise overwhelm one partition. These controls should preserve important traffic first and reject or delay work that can safely wait. The correct policy depends on whether the key represents a counter, inventory, a payment state, or another domain with different consistency requirements. Choose the Tradeoff Explicitly Every hot key mitigation trades one property for another. Buckets increase read work and make aggregate snapshots non-atomic. Application caching introduces staleness. Replication increases write cost and requires reconciliation. Queues delay visibility. These tradeoffs are often preferable to letting one shard become a single point of congestion. The right design starts with the semantics of the key. Decide whether the value must be exact, how long it can be stale, whether writes can be delayed, and what happens when the shared store is unavailable. Then measure the key’s read rate, write rate, payload size, and partition utilization. A mitigation that is safe for a view counter can be incorrect for inventory or payment state. Hot keys are a normal condition in systems with uneven demand. Monitoring should detect them early, and the architecture should provide a way to redistribute load before one partition limits the entire service. With bucketed writes, local caching, replication, queues, and admission control, teams can turn a concentrated source of traffic into a managed capacity decision instead of a cluster-wide incident.
AI has dramatically reduced the cost of generating code. But software engineering isn't just code generation. As implementation gets faster, the real bottlenecks move to planning, architecture, review, testing, deployment, and long-term maintenance. Teams that continue optimizing for coding speed alone will find themselves spending more time fixing problems they could have prevented. Speed Changes Where Engineering Time Goes Historically, the costs related to change and software rebuilds were massive. However, AI allows organizations to move incredibly fast. By orders of magnitude, engineering previously required more effort than any other SDLC phase. Yet, AI has created a reality where engineers can build quickly, lowering some of these costs of change. Unfortunately, building quickly in the wrong direction, based on the wrong conclusions, can fail to meet business needs. When engineers jump straight into coding, they often eliminate or reduce the time spent on: Planning: The team should know which problem it’s solving, what evidence supports the project, who makes the final decisions, and how it’ll measure success.Design: Show how the change fits into the existing system by identifying the services, interfaces, data handoffs, and failure scenarios the implementation must account for. As organizations fill the funnel with decisions that have no basis in fact, they send engineers down a path with vague requirements or a limited understanding of the business problem they face. Engineers fail to solve a problem or align to an area of need because the initial planning and design phases never occurred or failed to clearly outline the issue. Leadership sends the project back to the engineers, hoping they guess the second, third, or fourth time. Through this process, the bottleneck shifts upstream to planning and design, which puts pressure on the downstream stabilizing phases of testing, deployment, and maintenance. And it shifts in both directions. Pressure on planning, research, and experiment design is present. There is also pressure on quality and user acceptance. Think of it as creating scarcity around taste and judgment. The design, product, and business alignment conversations have yet to be accelerated by AI to the same degree as development. Finding value and aligning with the business are becoming the new bottleneck. Organizations have balanced those functions against historically long development cycles that need to be reconsidered before they further impact growth. Architecture, documentation, and the establishment of real, measurable goals are all valuable parts of the software development life cycle. There is an opportunity to turn this bottleneck into an additional accelerator if done the right way. The Costs of AI Hide in Additional Review Cycles and Repeated Fixes Before AI, software engineering teams could spend up to two months on discovery, communicating with stakeholders and mapping out the project. The deliverable was a report that represented the planning and design phases of the SDLC. Although not part of the final product, these reports were integral to its success. With AI accelerating the code implementation phase of the SDLC, the high engineering cost that drove a careful process to ensure the product answered the business question no longer exists. This single change in the SDLC has impacted the rest of the phases and made an evidence-based planning and design process more important. Planning and Design: Evidence-Based With Human-in-the-Loop AI can draft a document containing requirements and acceptance criteria. However, it can’t verify that the specification reflects actual users, system behavior, regulatory obligations, or operational limits. In reality, these documents lack an actual understanding of the organization's users, market constraints, or tradeoffs that prove a feature’s value. They fail to account for churn related to adding friction to processes. For example, customers may not be leaving because a retailer fails to apply discounts but because they never receive purchases on time. At this stage, a human should validate requirements documents against real user research, usage data, customer conversations, and stakeholder sign-offs. This confirmation enables organizations to achieve the best of both AI automation speed and the manual work that used to take two months. Testing Against Intended Behavior Tests need something outside the implementation to check against. That may be an approved requirement, acceptance criteria, an API contract, or known production behavior. When the same incomplete prompt produces both the code and its tests, the model will carry one mistaken interpretation into both. The tests may pass because they were written to expect the same incorrect behavior. Therefore, a faulty implementation can pass a generated test that treats the same incorrect behavior as expected. While engineers often write code, the gap between writing code and writing its tests creates a natural pause that often serves as a moment of reflection. Reviews must still occur to compare the code in the output against requirements, architecture decisions, and known failure modes. Deployment Requires Production Context A deployment configuration can work as expected in development or staging and still be incomplete for production. One common gap involves readiness checks. If traffic reaches a new version before the application is ready to receive requests, users may encounter timeouts, failed requests, or partially loaded pages. The configuration may also omit rollback criteria or acceptable downtime limits required during a release. Deployment configurations definitely require production context, but are also context themselves. Change management and attribution require some organizations to have a human flip the switch that sends things to production. This is an excellent opportunity to leverage the existing DevOps and Release Engineering principles we have built without AI. Approvals have long been a mix of people and code working together as part of CICD systems. Human-in-the-loop (HITL) workflows are the quick fix to many of these issues. Automated rollbacks and eval steps can help AI with this as well. However, without training the model on operational context, the AI has little to no information to extract and has to use a set of generic best practices that may not apply to the organization’s application. Maintenance: Preserving Institutional Memory Fragile code, like a retry block or an odd null check, often looks unnecessary but exists because of a previous incident. Whether it's legacy or AI-generated, this fix lacks documentation unless someone told the AI to document the change when it was made or an engineer documented it. When AI “cleans up” code and optimizes for visible logic, it removes this code, strips out the fix, reintroduces the bug, and forces the team to apply a new fix to resolve the problem. AI Is Changing Jobs, But Not the Way You Think Junior engineers may spend less time writing basic code and more time evaluating AI output. That review depends on experience with system behavior, architecture decisions, and production failures that they are still developing. For experienced engineers, technical review and ongoing maintenance activities have become the norm. In a recent survey, 52% of engineers said that most friction came from fixing subtle bugs in AI code, indicating that this shift toward HITL is already being felt in the workplace. As AI handles more implementation work, engineers spend more time defining the problem, choosing the approach, and verifying the result. While AI can accelerate execution, it still needs people to make the decisions around value or risk, creating a new type of bottleneck in software engineering.
DB changes can be streamed in real time via Oracle GoldenGate. Java applications can then consume them to validate/act on/route events without polling the DB. Goal Listen to real-time transactions in the DB, act on all operations within transactions, and maintain transaction boundaries with near real-time SLAs. Component Details 1. Source DB This is the source DB from which the Java service listens for transaction changes in real time. It can be any DB supported by the GoldenGate service (e.g., MySql/ Oracle). 2. OCI Source GoldenGate Service Deployment Input: DB Redo logsOutput: Trail file The GoldenGate service has two primary components: Extract and Replicat. Extract reads changes from the source DB in the form of redo and archive logs. It commits each transaction's DML operations until either COMMIT or ROLLBACK is identified. If it encounters a COMMIT, it stores the transaction in TRAIL files. If ROLLBACK is encountered, it discards those changes. The output of this deployment is a TRAIL file with an ordered list of all transactions that occurred in the source DB. 3. OCI Target GoldenGate Big Data Deployment Input: Trail files generated by traditional GG deploymentOutput: Ordered stream of operations within a transaction This is another deployment type that can be used for transaction management. We can use the DATA STREAMS feature instead, but this uses the AsyncAPI publish/subscribe interface, whereas big data deployment is a managed integration runtime for CDC delivery to Kafka. 4. Streaming Service Input: Trail files generated by traditional GG deploymentOutput: Ordered stream of operations within a transaction Ideally, Kafka keeps up with the scale of GoldenGate and downstream scalability. Kafka provides an option to add partitions while the partitions are receiving data. This feature is particularly useful for handling traffic bursts or batch SQL jobs on the source DB. 5. Java Service Kubernetes Deployment Input – Kafka messages, in this case individual operationsOutput – Application logic A Java application instance that can scale horizontally depending on the Kafka partition backlog. Use Case Java service is interested in listening to the entire transaction in the order that it is happening for a predefined set of tables.It does not want to listen to all the source table changesIt also does not want all columns of the changed row. It is interested in listening to only specific set of columns from given list of tables. Component Setup The detailed setup guide here can be used as general guidance, but I will focus on the specific changes needed to achieve the use case above. 1. Source DB Here we want to listen to redo logs from the source DB. To enable supplemental logging on the source DB, use the command below. SQL ALTER DATABASE ADD SUPPLEMENTAL LOG DATA In our use case, we also want to make sure we always get only the columns from the table that the Java service is interested in, and not all the columns. To enable table-specific supplemental logs, use the command below. SQL ALTER TABLE HR.EMPLOYEE ADD SUPPLEMENTAL LOG GROUP emp_group_loc (EMPLOYEE_ID, EMP_NAME, EMP_DEPARTMENT_ID); 2. OCI Source GG Deployment In this step, OCI source GG deployment is configured to listen to redo logs of the source DB. As per our use case the important step is to configure only required tables in Extract's parameter file. The list of tables basically makes sure transactions happening on only those tables will be identified by GG and will be collected in trail files. You can use this article to set up Extract. Example of parameter file for extract: SQL table HR.EMPLOYEE; Along with extract, a distribution path needs to be created within the same source deployment. This distribution path makes sure the target big data deployment can access the trail file records easily. The details to create the distribution path can be found in TASK 3 of the article. 3. OCI Target GoldenGate Big Data Deployment In this deployment, a process called 'replicat' is created; replicat reads transaction data from the distribution path created in step 2 above, then publishes those transactions. Replicat works using two configuration files. Parameter File Similar to Extract, Replicat also has a parameter file. This file contains mapping information on how operations from the source DB map to the target schema and table. Since we are not replicating transactions in the DB, a JSON with the target table mapped in the parameter file gets written to kafka topic instead. Example for a complete setup guide: use this guide. SQL MAP HR.EMPLOYEE, TARGET *.*; Properties File Three important parameters that decide how the source transactions get written to the target are 1. TopicMappingTemplate – Tells replicat about the target, where source transactions will be written, 2. opMode – How the source transaction will be written. Complete transaction or one operation at a time.3. keyMappingTemplate – This parameter is the hashing key used by the replicat Kafka producer. For all the parameters that are available in the properties file, please refer to section "11.2.10.1.3.2.2 Kafka Handler Configuration" in this guide. 4. Streaming Service and Java Consumer Service Since GG hashes based on transaction ID, it is guaranteed that all operations in a transaction will always be in the same partition. The Java service needs a consumer group used by all pods. At bootstrap, all pods register their client ID with the Kafka broker and reserve a partition. This mechanism makes sure a single pod always listens to one partition until a rebalance happens due to a pod going down or other reasons. This setup ensures the Kafka consumer in the Java service can poll and collect transactions while maintaining operation boundaries and sequence for downstream application logic within the same service. Separating polling logic from processing logic in the Java service helps avoid various timeout scenarios. We can dive deep into this mechanism in a follow-up article. Summary With this setup, we can guarantee that transactions at the source DB are accessed by a Java service in order and can run within a single transaction at a time. This workflow is useful for industries that need an eventing mechanism, such as fraud detection and financial applications, telecommunications and real-time billing, Retail and real-time inventory management, and AI and semantic search pipelines.
Every year Apple’s “What’s new in Swift” session drops thirty minutes of language features, and every year the recaps list them in the order they appeared on stage. That order is optimized for a keynote, not for your sprint. This one is sorted differently. The question for every feature is the same: how far is it from a diff I can open tomorrow? Some of these are one-line changes that delete a hack you’ve been apologizing for since 2021. Others are powerful, well-designed, and will not touch your app code for a year. Both deserve to be understood. Only one deserves a ticket. One thing to fix in your mental model before we start: this year’s session covers two releases, Swift 6.3 and 6.4. Some of what follows is already in your toolchain. Some is waiting for the next Xcode. I’ll flag which is which where Apple did. Each feature gets three lines: Lands in, Reach for it when, and a Verdict. Let’s go. Tier 1: Monday-Morning Changes 1. XCTest and Swift Testing Finally Speak the Same Language Lands in: Swift 6.4 If you’ve stalled on migrating to Swift Testing, it was almost certainly because of this: every test target has a pile of helpers built on XCTAssert*, and rewriting them all at once was never going to get prioritized. That blocker is gone. XCTest assertion failures, when called from a Swift Testing test, are now reported as test issues. And it works the other way too: #expect runs correctly inside an XCTestCase. Swift // Written in 2019. Still fine. func assertValidCart(_ cart: Cart) { XCTAssertFalse(cart.items.isEmpty, "cart must not be empty") XCTAssertGreaterThan(cart.total, 0) } Swift // A brand-new Swift Testing test can call it directly. @Test func checkoutBuildsCart() throws { let cart = try CartBuilder() .add(.sku("SNK-001"), quantity: 2) .build() assertValidCart(cart) // XCTest assertion, reported as an issue #expect(cart.total == 3_998) // Swift Testing expectation }// And the reverse: a legacy XCTestCase using #expect. final class LegacyCartTests: XCTestCase { func testWelcomeCouponApplies() throws { let cart = try CartBuilder() .add(.sku("SNK-001")) .apply(coupon: "WELCOME10") .build() #expect(cart.discount == 0.10) } } The practical consequence: you can write one set of assertion helpers and have them behave identically regardless of which runner calls them. Migration becomes a per-file decision instead of a per-target big bang. Reach for it when: You have more than a handful of XCTest-based helpers and want to move to Swift Testing incrementally. Verdict: Ship it. This is the single highest-ROI change in the session for any team with a real test suite. There’s a trap here, though. See the pitfalls section before you flip the switch. 2. @diagnose: Warnings You Control Per Declaration, Not Per Target Lands in: The 6.3/6.4 cycle Deprecation warnings have a lifecycle problem. The API gets deprecated. You can’t migrate yet, because the replacement is a different shape and the migration is a quarter of the work. So you either live with yellow triangles in the build log until everyone stops reading them, or you disable the warning for the whole module and lose it forever. @diagnose gives you a third option: change how a specific warning group behaves inside a specific declaration. Swift // Silence a known deprecation in exactly one place, with a paper trail. @diagnose(DeprecatedDeclaration, as: ignored, reason: "PaymentsKit v3 lands next quarter, tracked in PAY-4821") func tokenize(_ card: Card) -> Token { LegacyPayments.tokenize(card) // deprecated, and we know } Swift // Turn ON a check that's off by default, only where it matters. @diagnose(StrictMemorySafety, as: warning) func decodeFrame(_ bytes: UnsafeRawBufferPointer) -> Frame { // every unsafe access here now gets audited by the compiler }// Promote future errors to errors today, so nobody adds new instances. @diagnose(ErrorInFutureSwiftVersion, as: error) func parsePrice(_ raw: String) -> Decimal { // ... } Three modes, three very different uses: ignored for scoped suppression, warning for opting into stricter checks in security-sensitive code, error for ratcheting a migration forward one function at a time. Reach for it when: You’re mid-migration to Swift 6 language mode, or you have a security boundary you’d like the compiler to be paranoid about. Verdict: Ship it. Then add a lint rule that rejects as: ignored without a reason:. That parameter exists for a reason of its own. 3. Concurrency Stops Nagging You About the Wrong Things Lands in: Swift 6.4 This is a bundle of small fixes, and each one deletes a workaround you’ve probably written. Silently swallowed Task errors now warn. This one catches real bugs. A fire-and-forget task whose closure throws would just... drop the error. Now the compiler tells you. Swift // Before 6.4: compiles, error vanishes into the void. // 6.4: warning — you're ignoring a thrown error. Task { try await syncEngine.push(pendingChanges) } Swift // Option A: handle it inside the task. Task { do { try await syncEngine.push(pendingChanges) } catch { logger.error("push failed: \(error)") } }// Option B: keep the handle and check later. let push = Task { try await syncEngine.push(pendingChanges) } defer { await telemetry.flush() } // async calls in defer are allowed now try await push.value Note the defer in Option B. Calling async functions from a defer block was previously forbidden. Not anymore. weak let exists, and it unblocks Sendable. Any class holding a weak var delegate couldn't be Sendable without @unchecked. That was a lie you told the compiler to make it stop talking. Now you can make the weak reference an immutable binding, and Sendable checking is fine with it. Swift // Before: a lie. final class ImagePrefetcher: @unchecked Sendable { weak var delegate: PrefetchDelegate? } Swift // After: honest. final class ImagePrefetcher: Sendable { weak let delegate: PrefetchDelegate? init(delegate: PrefetchDelegate?) { self.delegate = delegate } } The reference can still go nil when the delegate deallocates. What changed is that the binding is immutable, and that's what Sendable cares about. ~Sendable says the quiet part out loud. You can now declare a type explicitly non-Sendable. Subclasses are free to opt back in. Swift class Screen: ~Sendable { } final class SplashScreen: Screen, @unchecked Sendable { } A second memberwise initializer for mixed-visibility structs. A struct with internal stored properties and a private one with a default used to get a single private memberwise init, which meant hand-writing the public one. Now the compiler generates both. Swift struct OrderDraft { var items: [LineItem] var address: Address private var validationCache: [String: Bool] = [:] } Swift // From another file in the same module — this now just works: let draft = OrderDraft(items: items, address: address) And some/any next to an optional no longer needs parentheses. Small. Satisfying. Reach for it when: You grep for @unchecked Sendable and find more than zero results. Verdict: Ship it. Run the 6.4 compiler over your codebase and treat every new “ignored error” warning as a bug report you didn’t have to write. 4. withTaskCancellationShield: Finish the Write Before You Bail Lands in: The 6.3/6.4 cycle (standard library) Checking Task.isCancelled before expensive work is the right default. Except for the moment when you're halfway through writing a file, committing a transaction, or flushing an analytics journal. Bailing there doesn't save work. It corrupts state. Inside the shield, every cancellation check reads false, all the way down the call stack. Swift extension AnalyticsJournal { func flush(_ batch: [Event]) { // Everything before this point may honor cancellation. withTaskCancellationShield { // Inside: Task.isCancelled == false, so lower layers // won't abort mid-write. storage.append(batch) storage.commit() } } } Apple’s guidance is worth repeating: keep the shielded region short. This is for “finish or roll back what I already started,” not for “ignore cancellation because it’s inconvenient.” Reach for it when: You have a write path that’s already had a bug caused by cancellation landing between two dependent operations. Verdict: Ship it, surgically. One shield per write path, none anywhere else. 5. ProgressManager: Progress for the Async/Await Era Lands in: The 6.3/6.4 cycle (Foundation) Foundation’s old Progress was designed around KVO and implicit parent/child registration via thread-local state, which is exactly the kind of thing that stops making sense once your code is structured concurrency. ProgressManager separates composition (who owns which share of the work) from reporting (who's watching), and it does it with types instead of magic. Swift let manager = ProgressManager(totalCount: 100) Swift // Hand out a slice of the total to a child operation. try await exporter.exportOrders(manager.subprogress(assigningCount: 100))extension OrderExporter { func exportOrders(_ progress: consuming Subprogress? = nil) async throws { let steps = progress?.start(totalCount: 3) try await fetch(); steps?.complete(count: 1) try await encode(); steps?.complete(count: 1) try await upload(); steps?.complete(count: 1) } }// Observe from the UI side. Task { for await fraction in Observations({ manager.fractionCompleted }) { exportButton.title = "Exporting \(Int(fraction * 100))%" } } Two details to notice. The child function takes consuming Subprogress?, so the caller can pass nil when nobody's watching and the function doesn't care. And the manager supports typed metadata, so you can attach domain-specific values (bytes transferred, items processed) and summarize them across the tree without stringly-typed user-info dictionaries. Reach for it when: Any multi-stage async flow that shows a progress bar. Uploads, exports, media processing, initial sync. Verdict: Ship it for new flows. Don’t rewrite working Progress code just to use it. 6. anyAppleOS: One Availability Line Instead of Five Lands in: The 6.3/6.4 cycle Apple aligned OS version numbers last year (everything is 26, then 27). This year Swift cashes that in: a single platform name covers all of them. Swift // Before @available(macOS 27, iOS 27, watchOS 27, tvOS 27, visionOS 27, *) func showLiveActivityGlance() { } Swift // After @available(anyAppleOS 27, *) func showLiveActivityGlance() { }// Set the default, then carve out exceptions. @available(anyAppleOS 27, *) @available(tvOS, unavailable) func startCheckoutHandoff() { }// Works in conditional compilation too. #if os(anyAppleOS) import WidgetKit #endif Reach for it when: You ship a framework or app extension across three or more Apple platforms. Verdict: Ship it if your minimum versions line up across platforms. If they don’t, this does nothing for you. 7. Module Selectors (::): Name Collisions, Resolved for Real Lands in: Swift 6.3 Two modules export a type with the same name. You disambiguate with ModuleName.TypeName. This works until the module also contains a type with the same name as the module, at which point Swift prefers the type, looks inside it, finds nothing, and errors. The new :: syntax makes the left side always a module name. Swift import SwiftUI import DesignSystem // also exports a `View` type Swift struct ProductCard: SwiftUI::View { var body: some SwiftUI::View { DesignSystem::Card { /* ... */ } } } It works on members too, which matters when two modules add same-named methods to one type via extensions: Swift // Module Fulfillment adds `cancel()` to Order. // Module Payments also adds `cancel()` to Order. order.Payments::cancel() // refund, don't un-ship Apple’s own note on this: it’s for collisions between modules you don’t control, and for macro expansions where you can’t predict what else got imported. Do not design APIs that rely on it. Reach for it when: The View-vs-View problem, or you write macros. Verdict: Ship it as a fix, never as a design. Tier 2: The Testing Toolbox Got Sharper Lands in: Swift 6.4 Beyond the XCTest bridge, Swift Testing picked up three things that address real CI pain: Swift @Test(arguments: Storefront.all) func priceFormatting(storefront: Storefront) throws { // Skip *this argument*, not the whole test. if storefront.currency == nil { try Test.cancel("no currency configured for \(storefront.id)") } Swift let text = PriceFormatter(storefront: storefront).string(from: 1_999) // Worth a look, not worth blocking the merge queue. if text.count > 12 { Issue.record("\(storefront.id) renders unusually long", severity: .warning) } #expect(text.contains("19")) } Test.cancel is the interesting one in parameterized tests: you cancel individual arguments instead of either running them to a meaningless completion or failing the whole test. Issue.record(severity: .warning) surfaces things you want a human to notice without turning the pipeline red. And swift test gained a repeat mode: rerun until pass or until fail, with a cap on repetitions. In "until pass" mode only the failing tests rerun. Flaky-test hunting, built in. Verdict: Ship it. Test.cancel alone will clean up a lot of guard ... else { return } at the top of parameterized tests. Tier 3: Performance, With Receipts Everything here is real, well-designed, and will not show up in most app code this year. It will show up in the frameworks your app depends on. Know it exists; reach for it when the profiler tells you to. 8. borrow/mutate Accessors: Computed Properties Without the Copy Lands in: Swift 6.4 A computed property with get/set moves the value by copy. For an Int, who cares. For an InlineArray of 256 Ints, that's a two-kilobyte struct copied out and back just to touch one element. borrow gives read access to the storage in place. mutate gives exclusive write access in place. Swift struct FrameBuffer: ~Copyable { private let storage: UnsafeMutablePointer<[1024 of UInt8]> Swift var pixels: [1024 of UInt8] { borrow { storage.pointee } mutate { &storage.pointee } } }buffer.pixels[512] = 0xFF // in place. No 1 KB round trip. As a bonus, the property can now hold non-copyable values, which get/set fundamentally couldn't. Verdict: Reach for it in wrappers around large value types. This is the most practical thing in the ownership block. 9. Continuation: Resume-Once, Checked at Compile Time Lands in: The 6.3/6.4 cycle (standard library) Every codebase bridging callbacks to async has the same two options: CheckedContinuation (safe, runtime cost) or UnsafeContinuation (fast, and one double-resume away from a crash in production). The new Continuation type verifies at compile time that you resume it exactly once. Safer than checked, as cheap as unsafe. The session names the type and the guarantee without walking through the call site, so check the standard library docs for the exact spelling before you write it in a code review. Verdict: Adopt it in new bridging code. Sweep the old UnsafeContinuation sites when you have a quiet week. 10. MutableRef: Hoist the Dictionary Lookup Out of the Loop Lands in: The 6.3/6.4 cycle (standard library) You have a loop that increments counts[key] on every iteration. The lookup is the same every time. Until now, holding a dictionary slot "open" across a loop required moving the loop into a nested function and passing the slot as inout. Obscure, and everyone who's found it has felt slightly dirty. Swift func countMentions(of tag: Tag, in posts: [Post], into counts: inout [Tag: Int]) { var slot = MutableRef(&counts[tag, default: 0]) // one lookup Swift for post in posts where post.tags.contains(tag) { slot.value += 1 // no re-hash } } Ref (read) and MutableRef (write) are non-escapable, so the compiler knows the access ends when the variable leaves scope. They can be stored, passed, returned, and used in generics, which means APIs that hand out a live reference to one of their properties are now expressible. Verdict: Reach for it in hot loops the profiler has already pointed at. 11. Iterable: for Loops That Borrow Instead of Copy Lands in: Swift 6.4 Sequence copies each element out to the loop variable. Iterable lets the loop borrow them, in batches, via Span. No retain/release for reference types or copy-on-write containers, non-copyable elements are supported, and the loop can throw. The constraints matter: you can’t mutate the collection while iterating (exclusivity is enforced), and for prefers Sequence when a type conforms to both. So adopting Iterable on a type that already has Sequence changes nothing. Verdict: Know it exists. Author it in collection-heavy library code. Don’t expect free speedups elsewhere. 12. @inline(always) and @specialized: Telling the Optimizer What You Know Lands in: @inline(always) in 6.4, @specialized in 6.3 @inline(never) has existed forever. It now has a twin that forces inlining even when the optimizer's heuristics say no. For class methods, pair it with final or dynamic dispatch will still get in the way. @specialized asks the compiler to emit a concrete version of a generic function for a type you know is hot, which matters most in libraries where the compiler can't see the call sites. Swift @inline(always) func clamp01(_ x: Float) -> Float { min(max(x, 0), 1) } Swift @specialized(where Bytes == [UInt8]) func checksum<Bytes: Sequence<UInt8>>(_ bytes: Bytes) -> UInt32 { // ... } Verdict: Measure first. Both can make a binary bigger and slower when applied on a hunch. The Traps Nobody Puts in the Recap The XCTest bridge reports failures as warnings by default. An XCTAssert that fails inside a Swift Testing test does not fail the test until you opt in via the Xcode build settings. Flip that switch on day one, or you'll have "green" runs that are quietly losing signal.@diagnose(..., as: ignored) is debt with a nicer syntax. Require reason: in code review. Grep for it quarterly.@inline(always) and @specialized are not free. Apple describes them as tools you "usually won't need." Believe that. Profile, apply, re-profile.Iterable won't replace Sequence automatically. If your type has both, for picks Sequence. No silent speedup.weak let doesn't make the reference permanent. It can still go nil. What you gain is that the immutable binding no longer blocks Sendable.anyAppleOS requires aligned minimums. Different minimum versions per platform means you're back to the long form. What I’d Do This Week Open a branch. Compile with 6.4. Fix every new “ignored thrown error” warning. That’s your free bug sweep.Replace @unchecked Sendable on delegate-holding classes with weak let.Turn on XCTest-failure promotion in Swift Testing, then migrate one test file.Put a cancellation shield around your one scariest write path.Bookmark borrow/mutate and MutableRef for the next time Instruments shows you a copy you didn't ask for. Everything else in this session (Swift-to-C via @C, the official Android SDK, Wasm at 35–40x faster bridging, embedded Swift growing its subset) is genuinely exciting and belongs in a different article, one about where Swift is going rather than what lands in your app.
The more data companies gather on their customers, products, and code, the more duplicated information accumulates, confusing the teams who depend on it. You don't go through millions of records manually. So deduplication is now a machine learning problem. This article looks at the current state of duplicate detection in ML systems where the old techniques still work, and what embeddings and large language models have brought to the table. How Do ML Systems Detect Duplicates? It is possible for a person to look at two pictures or two strings and know if they are the same. Teaching a machine to do the same raises difficult questions. What is 'like'? There are degrees of resemblance. Is “Jon Smith, 12 Oak St.” the same customer as “Jonathan Smith, 12 Oak Street?” In practice, modern deduplication pipelines work like a funnel that begins with cheap, fast methods and introduces intelligence (and cost) only where necessary: Normalize block. Clean the data (case, punctuation, abbreviations, etc.) and group records into blocks where they share something in common (e.g., postal code or the first letters of a name). Only records within the same block are compared.Score candidate pairs using string metrics. Fast deterministic similarity measures catch typos, transpositions, and formatting differences.Embeddings for meaning. Dense vectors and approximate nearest neighbor search can discover pairs that are worded differently but mean the same.Push borderline cases up. If the earlier stages cannot be confidently settled, an LLM or a human reviewer resolves the pairs. Each stage decreases the number of pairs that the next, more expensive stage has to consider. This makes the approach affordable at scale. String Metrics Still Matter String metrics are the first line of defense. They are fast, interpretable, and do not need a model. Some common options are: Hamming distance: The number of positions at which two strings of equal length differ, or the number of substitutions needed to change one string into the other.Levenshtein distance: Generalizes Hamming to allow insertions and deletions as well as substitutions, so it works on strings of different lengths.Jaro-Winkler similarity: Measures the number of matched characters and transpositions, with more weight for a common prefix. It works particularly well with short strings, such as personal names.Sørensen–Dice coefficient: A measure of similarity based on the number of common bigrams (adjacent letter pairs).Jaccard similarity and TF-IDF cosine similarity: Comparing sets of tokens or weighted vectors of words; the latter is more suitable for longer text.MinHash with locality-sensitive hashing (LSH): a cheap way to estimate set similarity, so you can find near-duplicate documents without comparing them all.Learnable distance: understands that different domains have different weights for edit operations and learns those weights from labeled data. In a modern pipeline, these scores are typically features that go into a classifier, not the final answer. For example, a model can learn that a mismatch in a phone number is much more important than a mismatch in a middle initial. Vector Search and Embeddings String metrics break down when two texts say the same thing in different words. Take two questions from a Q&A site. How can I get cheap hotel rooms?What are the best ways to get cheap hotel deals? They have very few words in common, and thus have a rather high edit distance rating. Sentence-embedding models, by contrast, map each text into a vector space where similar texts are near each other. Then finding duplicates becomes a nearest-neighbor search: Python from sentence_transformers import SentenceTransformer import faiss questions = [ "How do I book cheap hotel rooms?", "What are the best ways to find cheap hotel deals?", "How do I renew my passport?", ] model = SentenceTransformer("all-MiniLM-L6-v2") vectors = model.encode(questions, normalize_embeddings=True) # Inner product on normalized vectors equals cosine similarity index = faiss.IndexFlatIP(vectors.shape[1]) index.add(vectors) scores, neighbors = index.search(vectors, k=2) for i, (row_scores, row_ids) in enumerate(zip(scores, neighbors)): for score, j in zip(row_scores, row_ids): if j != i and score > 0.7: # tune this threshold on labeled data print(f"Possible duplicate: {questions[i]!r} ~ {questions[j]!r} ({score:.2f})") The flat index above is fine for thousands of records. For millions, switch to an approximate index such as HNSW or IVF, which trades a little recall for large speedups. Before reaching for approximate search, check whether simple blocking already shrinks your comparison space enough. The similarity threshold is not universal. Tune it on a labeled sample of your own data, because the right cutoff for support tickets will differ from the right cutoff for customer names. LLMs for the Tough Cases Embeddings provide you with candidates, not certainty. Two product listings might differ only in a model number that matters, but otherwise be nearly the same vector. That’s where big language models are earning their place now, as a pair resolver that looks at two candidate records and decides whether they refer to the same real-world thing. Open-source tooling has started to package this pattern. One example is cocoindex, which uses FAISS embedding similarity to find near-duplicate entity names, and then queries an LLM-based resolver to confirm matches and choose the better canonical name. Academic work is moving in the same direction. A SIGMOD 2026 paper on in-context, clustering-based entity resolution with LLMs includes ways to reduce hallucinations and API calls. There are two caveats: Cost and delay. Send LLM calls only for the pairs the cheaper stages couldn't resolve, as LLM calls are much more expensive than string comparison. That’s why the funnel is there.Matches that have been hallucinated. An LLM can confidently speak of a wrong merge. Keep confidence scores, log reasoning, and have a human in the loop for high-stakes merges. Real World Examples Questions on Q&A Sites: Duplicates Sites such as Quora and Reddit are constantly bombarded with the same questions over and over again because users don't know someone else has already asked them. The most famous benchmark here is the Quora Question Pairs dataset, which came out in 2017 with about 404,000 labeled training pairs [check exact counts; the original paper reported 404,290 training pairs and 2,345,795 test pairs]. The benchmark is hard because duplicates are meaning-based, not word-based; capitalization, abbreviations, and context all matter. The above hotel example is a good first test for any approach. Embedding models do a much better job on this task than edit distance. Marketplace Duplicate Listings On classified-ad and marketplace sites, sellers will often re-post or slightly edit a listing if it's not performing. The same funnel applies: block by category and location, score titles and descriptions with string metrics, use embeddings to catch rewording, and flag the clusters for removal or consolidation. Another signal could be added for image similarity when listings have photos. Copy Code The biggest example of how much duplication builds up at scale is GitHub. Its Octoverse 2025 report says the platform now has 630 million total repositories, with 121 million added in 2025 alone, and 395 million of them are public or open source. It has more than 180 million developers. At that size, duplicated code is a must. Code duplication can be exact copies or clones that behave the same, but are written differently. Research on the topic (including a widely cited 2017 study of code duplication on GitHub, DéjàVu) found that a large share of JavaScript files were copies [verify the study’s details and the 93% figure before citing it, and make clear that this was academic research rather than something GitHub itself runs]. The newer story is AI-assisted coding. GitClear research of 211 million changed lines of code in 2025 found an eightfold increase in 2024 in code blocks that duplicate adjacent code with five or more lines. It also found for the first time that copy-pasted lines outnumbered moved (refactored) lines. GitClear sells code-review analytics, and its data shows correlation with AI adoption, not proof of cause, so treat it as vendor research. But it hints that code deduplication will grow in importance as AI tools make it easier to generate new code than to consolidate it. Duplicate Records in CRM Software Traditional CRM de-duplication relies on matching rules, and rules can’t cover all fuzzy cases. A learning-based approach, however, identifies fuzzy duplicates by combining multiple string metrics and learning from human feedback: the system shows you a pair of records, you mark them as duplicate or distinct, and it adjusts its matching accordingly. You can learn more about this in my previous article. The second big advantage is scalability. Comparing every record with every other record is quadratic. Now suppose you have 50,000 records and add 5,000 new records. Which means 50,000 x 5,000 = 250,000,000 comparisons to compare each new record with each existing record. That is almost 7 hours, 10,000 comparisons per second: 250 000 000 / 10000 / 3600 ≈ 6.94 hours If you compare every record with every other record in the combined 55,000 records, you get roughly 1.5 billion pairs or about 42 hours at the same speed. Blocking does this by grouping records with common characteristics and only comparing within each group, which reduces the amount of work to be done dramatically. Vector search does something similar, returning only the nearest neighbors of each record. AI Pipelines Duplicates Deduplication has also become a core step in the artificial intelligence systems themselves: Training data: Removing near-duplicate documents, often using MinHash and LSH, so models don't over-weight repeated text.RAG corpora: Collapsing repeated passages so retrieval doesn’t fill the context window with copies of the same content.Knowledge graphs: Marrying differently named entities like “Mr. Edwards,” “Brad Edwards,” and “Detective Edwards,” so relationships don’t break across duplicate nodes. Some tools now leverage a combination of deterministic rules, semantic similarity, LLM reasoning, and human review to do this.Pitfalls to plan for: False merges are worse than missed duplicates. Merging two different customers can corrupt the records and break downstream processes. Choose the more costly error and set the limits accordingly Just measure it. Evaluate on a labeled sample of pairs with precision, recall, and F1. Re-check as your data changes. Keep Humans in the Loop Low-confidence pairs are passed to reviewers, and their decisions are fed back into the system. A few considerations remain important: Data privacy: Records about people often contain sensitive information. Confirm what you send to hosted embedding or LLM services, and apply your organization's data-handling policies.Provenance: Keep original records and a merge log so that a bad merge can be undone. Machine learning will play an increasingly important role in deduplication, but it doesn't eliminate the need for human judgment. Any CRM, database, or codebase has a real price for duplicate data, and hand-written filters cannot handle the variety of ways duplicates show up. What the last few years have taught us in practice is not that one technique has replaced another. It’s that the best systems layer them: string metrics and blocking for speed, embeddings for meaning, and LLMs or people for the hardest calls. Start with the cheapest stage that works on your data, evaluate it, and only add the next stage if the results justify the cost.
Abstract Public statistical datasets rarely fail with an error. When something goes wrong, they hand back a plausible number, an HTML page with a success status, or a series that has quietly stopped updating. This article describes the ingestion pipeline behind a residential concrete cost estimator that combines four public datasets: an annual occupational wage survey, a monthly producer price index, a 2022 industry benchmark of material prices, and a ZIP code to metro area crosswalk. For each dataset, I document the failure mode we actually observed, the evidence for it, and the TypeScript code that turns it into an explicit state. The central claim is that every silent failure should become a typed value that travels with the number all the way to the user interface. The pipeline runs on a local CLI and a scheduled Cloudflare Worker, writes to Cloudflare D1, and serves a static site. Introduction The application, Concrete Cost Calculator, estimates what a home concrete job costs in a given metro area. Labor is priced from hourly wages for cement masons and concrete finishers in 393 metropolitan areas. Material is priced from a 2022 national benchmark for ready-mix concrete, moved forward in time with a producer price index and adjusted for four broad regions. A visitor's ZIP code is mapped to a metro area through a crosswalk file. Each of these datasets is documented and free to use. Each one also failed us at least once in a way that produced no exception. The cost of such a failure is high because the output is a price: a wage parsed as zero, or a material index read six months stale, yields an estimate that looks reasonable and is wrong. The rest of the article follows the data from source to screen. Section 2 describes the architecture and why it is split the way it is. Section 3 covers five failure modes and their defenses. Section 4 describes how provenance is carried with every value. Section 5 covers the write path. Sections 6 and 7 discuss trade-offs and limitations. System Overview The pipeline has three entry points, and the split between them is driven by two hard constraints: memory and network reachability. Figure 1. Data flow from the four datasets to the static site. Memory. The wage survey is released once a year as an archive of spreadsheets of roughly 40 MB. Parsing the metro and nonmetro workbooks needed a heap of about 965 MB in Node. A Cloudflare Worker is limited to 128 MB of memory, so the annual parse runs in a local CLI and the result is written to D1 as SQL. Reachability. The three remote APIs sit behind three different CDNs and behave differently depending on where the request comes from. During development, we recorded the following: Dataset APIFrom the WorkerFrom a browserReasonZIP crosswalkblocked (nginx 403)worksthe API returns access-control-allow-origin: *Price indexworksfailsno CORS headers are sentIndustry benchmarkworksnot testedparameters must be packed into one codeset value The first two rules point in opposite directions, which is why no single fetch strategy covers all sources. The crosswalk is pulled quarterly through the CLI. The price index and the benchmark are pulled by a separate ingest Worker on a monthly cron (17 5 1 , 05:17 UTC on the first day of each month). The site itself is a different deployment with no custom Worker entry: every page is prerendered, the rate table is exported from D1 to a static JSON file at build time, and the calculator runs in the browser. The only server route maps a ZIP code to a metro area. Failure Modes and Defenses 1. A Numeric Cell Has Three States Claim. A spreadsheet cell from the wage survey can hold a number, a flag, or nothing, and these must stay distinct. Evidence. The survey publishes markers inside numeric columns. An asterisk means the wage estimate is unavailable, a double asterisk means the employment estimate is unavailable, and a hash means the value is above the top code. The common idiom Number(raw) || 0 converts "*" to NaN and then to zero, so "no wage available" becomes "this metro pays $0 an hour". Nothing throws. The comment above our parser states the rule directly: Never write Number(raw) || 0. Number('*') is NaN, and NaN || 0 silently turns "wage not available" into "hourly wage $0". An empty cell is a third state: "there is no cell" is different from "the cell is flagged as unavailable", and both are different from "the cell is 0". Defense. Parse into a value and a flag, and keep a third state for a truly empty cell. The parser does not assume the set of flags is fixed, since a future release may introduce a new symbol. TypeScript export interface NumericCell { value: number | null; /** Any non numeric marker such as '*', '**' or '#', kept verbatim. */ flag: string | null; } export function parseNumericCell(raw: unknown): NumericCell { if (raw === null || raw === undefined) return { value: null, flag: null }; if (typeof raw === 'number') { return Number.isFinite(raw) ? { value: raw, flag: null } : { value: null, flag: String(raw) }; } const trimmed = String(raw).trim(); if (trimmed === '') return { value: null, flag: null }; const asNumber = Number(trimmed); if (Number.isFinite(asNumber)) return { value: asNumber, flag: null }; // Anything else is a marker. Store it; never coerce it. return { value: null, flag: trimmed }; } The database schema mirrors this: each wage column is nullable and has a companion flag column. 2. A Successful Response Can Be a Block Page Claim. An HTTP status code is insufficient evidence that a response contains data. Evidence. The crosswalk API returned JSON on one day and an HTML 403 page two days later, for the same request and with no code change on our side. The main website of the price index publisher sits behind bot protection that rejects scripted clients even with a browser user agent, while its separate API host answers normally. A JSON parser fed an HTML page throws a confusing syntax error at best; a lenient CSV or table parser may return an empty result and let the run "succeed". Defense. Sniff the start of the body before parsing, and fail with an error that names the likely cause. TypeScript export function detectBlockedResponse(bodyText: string): string | null { const head = bodyText.slice(0, 1500); if (/^\s*</.test(head)) { if (/access denied/i.test(head)) return 'Access Denied (Akamai style block page)'; if (/cloudflare/i.test(head)) return 'Cloudflare block page'; if (/403 forbidden/i.test(head)) return 'nginx 403 block page'; return 'HTML response where JSON was expected'; } return null; } Both parsers in the ingest Worker, for the price index and for the benchmark, call this function before touching the body and throw a BlockedResponseError when it returns a reason. The run fails, nothing is written, and the previous data stays in place. 3. Price Indexes Cannot Be Compared by Their Levels Claim. Two price index series can only be compared through their own growth over the same window. Evidence. In January 2019, the national producer price index for ready mixed concrete stood at 277.4, while the four regional series stood between 121.0 and 127.5. Dividing a regional level by the national level gives a factor of roughly 0.45, which would suggest concrete in the South costs less than half the national price. The gap reflects different base periods, and even with identical base periods, the level of an index encodes growth relative to its own base. Levels from different series do not describe relative prices. Defense. Anchor material cost in a real dollar figure, then move it with growth ratios. The 2022 benchmark reports the national value of ready-mix concrete shipped and the quantity shipped. Their ratio is $110.02 per cubic yard. That anchor is moved forward with the ratio of annual average national index values. A region's factor is its own growth divided by national growth over the same window, so each series is only ever compared with itself. TypeScript export function yoyGrowth(points: readonly IndexPoint[], anchorYear: string, currentYear: string): GrowthResult { const anchorAverage = annualAverage(points, anchorYear); const currentAverage = annualAverage(points, currentYear); // partial year allowed const ratio = currentAverage / anchorAverage; return { anchorYear, currentYear, anchorAverage, currentAverage, ratio, pct: (ratio - 1) * 100 }; } export function extrapolateMaterialPerCuYd(baseline: MaterialBaseline, nationalPoints: readonly IndexPoint[], anchorYear: string, currentYear: string) { const growth = yoyGrowth(nationalPoints, anchorYear, currentYear); return { baseline, growth, value: baseline.pricePerCuYd * growth.ratio }; } export function computeRegionFactor(region: Region, regionPoints: readonly IndexPoint[], nationalPoints: readonly IndexPoint[], anchorYear: string, currentYear: string) { const regionGrowth = yoyGrowth(regionPoints, anchorYear, currentYear); const nationalGrowth = yoyGrowth(nationalPoints, anchorYear, currentYear); const freshness = assessFreshness(regionPoints); return { region, regionGrowth, nationalGrowth, factor: regionGrowth.ratio / nationalGrowth.ratio, freshness }; } With the 2022 annual average as anchor and the 2026 average through August as the current value, the national figure comes to $132.33 per cubic yard. That figure is the base of the national estimate for what ready-mix concrete costs per yard, which adds a 3 percent waste allowance and the regional factor. A unit test pins it. A second test asserts that the tempting alternative, annual average to latest month, does not reproduce it. The two methods differ by less than a percentage point here, and the test exists so that nobody switches methods silently and mixes two windows inside one pricing chain. 4. A Series Can Stop Updating Without Saying So Claim. A monthly series that repeats the same value for several months should be treated as possibly stale, and the decision about what to do with it should be explicit. Evidence. Over 92 months of data, the national series and the Midwest and South regional series changed value in 23 of the last 24 months, and their longest run of identical values was one or two months. The Northeast series held the same value for six consecutive months, March through August 2026, and changed only 9 times in 24 months. The West series held for five months, April through August 2026. Defense. Measure the longest flat run and compare it with a threshold placed between the two observed populations. Normal series topped out at two months; the anomalous ones started at five. A threshold of four months separates them with a margin on both sides. TypeScript export const FRESHNESS_STALE_THRESHOLD_MONTHS = 4; export function assessFreshness(points: readonly IndexPoint[]): FreshnessReport { if (points.length === 0) throw new Error('assessFreshness: empty series'); const sorted = [...points].sort((a, b) => pointYm(a).localeCompare(pointYm(b))); let longestFlatRunMonths = 1; let currentRun = 1; let flatRunRange: string | null = null; for (let i = 1; i < sorted.length; i++) { if (sorted[i].value === sorted[i - 1].value) { currentRun++; if (currentRun > longestFlatRunMonths) { longestFlatRunMonths = currentRun; flatRunRange = `${pointYm(sorted[i - currentRun + 1])}..${pointYm(sorted[i])}`; } } else { currentRun = 1; } } const last24 = sorted.slice(-24); let changesInLast24Months = 0; for (let i = 1; i < last24.length; i++) { if (last24[i].value !== last24[i - 1].value) changesInLast24Months++; } return { longestFlatRunMonths, flatRunRange, changesInLast24Months, isStale: longestFlatRunMonths >= FRESHNESS_STALE_THRESHOLD_MONTHS, }; } What to do with a stale series is a policy question. Rejecting it would leave the region with no material signal at all, because the benchmark is withheld below the national level for confidentiality (state queries come back as HTTP 204 with no body). The region would fall back to the national value, which is precisely the "national average with a local label" behavior the product exists to avoid. We therefore keep the value and attach a note to its provenance so the interface can say the source series has not moved. Figure 2 summarizes the decisions taken for each data point across sources. Figure 2. Per-data-point decisions, from response to stored row. 5. Coverage Gaps Need Explicit Fallbacks Claim. When a dataset does not cover every entity, the fallback should be ordered, recorded on the value, and loud when exhausted. Evidence. The wage survey reports wages for cement masons and concrete finishers in 336 of the 393 metro areas, which is 85.5 percent. The union of three related occupations covers all 393. The crosswalk has 47,629 rows for 39,472 distinct ZIP codes, because a ZIP that spans several metros is split into one row per metro, each with a share of addresses. Defense. For wages, try occupation codes in a fixed order and stop the build if none has a value. The fallback reason is stored on the value. In the cost model, a substitute wage is scaled by the median ratio between the two occupations across metros that report both (1.1373 for construction laborers), and its provenance level drops from sourced to calibrated. TypeScript export const TARGET_OCC_CODES = ['47-2051', '47-2061', '47-1011'] as const; export function resolveWage(area: string, rows: readonly MetroWageRow[]): WageResolution { for (const occCode of TARGET_OCC_CODES) { const row = rows.find((r) => r.occCode === occCode && r.hMean != null); if (row) return { occCode, hMean: row.hMean as number, hPct25: row.hPct25, hPct75: row.hPct75, totEmp: row.totEmp }; } // Coverage changed since it was last measured. Publishing a metro with no wage is worse than failing. throw new Error(`resolveWage: metro ${area} has no usable mean wage in ${TARGET_OCC_CODES.join(', ')}`); } For ZIP codes, every row is kept, and exactly one per ZIP is marked primary: the row with the largest share. Only three ZIP codes in the current file have a tie, and they are resolved by order of appearance, which is deterministic. Keeping the secondary rows costs little and leaves room for weighted pricing later. TypeScript export function markPrimaryRows(rows: ZipMetroRow[]): ZipMetroRow[] { const bestByZip = new Map<string, ZipMetroRow>(); for (const row of rows) { const current = bestByZip.get(row.zip); if (!current || (row.totRatio ?? -Infinity) > (current.totRatio ?? -Infinity)) { bestByZip.set(row.zip, row); } } const primarySet = new Set(bestByZip.values()); return rows.map((row) => ({ ...row, isPrimary: primarySet.has(row) })); } The national wage is also a derived figure. The survey's metro files contain no national row, so the national benchmark is the employment weighted mean across the 393 metros, and its provenance says so explicitly. Presenting it as a figure published by the survey would be inaccurate. Provenance Travels With the Value Every parameter in the cost model is a Tracked value: a number plus a provenance record. TypeScript export type ParamLevel = 'sourced' | 'calibrated' | 'assumed'; export type GeoLevel = 'metro' | 'state' | 'region' | 'national'; export interface Provenance { level: ParamLevel; source: string; // the dataset name ref?: string; // occupation code or series identifier asOf: string; // the period the data describes, such as '2025-05' geo?: GeoLevel; // the level at which the value actually applies fellBackFrom?: GeoLevel; // set whenever a fallback happened note?: string; // e.g. the substitute occupation or a stale series } export interface Tracked<T = number> { value: T; prov: Provenance; } The three levels have strict meanings. sourced means the value is a published figure or a mechanical function of published figures. calibrated means a multiplier or substitute chosen by us and checked against outside evidence. assumed means no public data exists and the value is our estimate. The levels exist partly for honesty and partly for compliance: the terms of use for the source data require that modified figures not be presented as if they came straight from the publisher. Figure 3 shows the result for a metro that uses the substitute occupation. The fallback notice appears next to the price, and each line of the breakdown carries its level. Figure 3. Provenance as rendered in the estimate breakdown. The Write Path Two properties matter on the write side: a run that changes nothing should write nothing, and every stored row should be traceable to its origin. The ingest Worker keeps material baselines as an append-only history. Before inserting, it loads the latest stored row for each series and keeps only rows whose period or value differ. TypeScript export function selectRowsToWrite( newRows: readonly MaterialBaselineRow[], existingLatest: ReadonlyMap<string, Pick<MaterialBaselineRow, 'period' | 'value'>>, ): MaterialBaselineRow[] { return newRows.filter((row) => { const existing = existingLatest.get(rowSeriesKey(row)); if (!existing) return true; // first time this series is seen return existing.period !== row.period || existing.value !== row.value; }); } This makes the monthly cron safe to run more often than the data changes. In practice, the regional series change less often than monthly, so many runs end with zero rows written, and the log line records how many were skipped. For traceability, the original wage archive is never stored in the database. Instead each row records the source file name, the publication date and the SHA 256 digest of the archive, which is enough to identify the exact input later. The CLI writes wage rows with INSERT OR REPLACE on a unique key of metro, occupation, and year, and replaces the crosswalk table as a whole because each crosswalk file is a complete snapshot. Rerunning either import produces the same table. One operational lesson belongs here as well. When syncing the local database to the remote one, a full export would have overwritten the remote material table, which held six rows produced by the Worker, with the local copy, which held none. Tables owned by different writers have to be pushed one at a time. Discussion Annotate or reject. The pipeline consistently keeps imperfect data and labels it, with one exception: it refuses to publish a metro with no usable wage at all. The reasoning is that a labeled approximation is more useful to the reader than a silent national fallback, as long as the label reaches the screen. When nothing usable exists, failing the build is the only honest option. Where to parse. Moving the wage survey parse out of the Worker costs some automation, since a person has to download the annual file. In exchange, the Worker stays small, and the heavy parse runs where memory is cheap. The annual cadence makes the manual step acceptable. Tests as specifications. Several of the decisions above are encoded as tests with real recorded responses: the $132.33 anchor, the rejection of the latest-month method, the staleness threshold, and the three-state cell parser. These tests document the reasoning as much as they guard against regressions. Limitations The wage data describes May of the reference year and is published about a year later, and it is a three-year rolling estimate. Material prices are national with four regional factors, a much coarser resolution than labor. The material anchor is from 2022 and relies on the price index for everything since. About 4 in 10 ZIP codes fall outside the 393 metro areas and receive national figures. The fixed setup cost in each estimate has no public data source and is labeled as an estimate. Conclusion Public statistical data is a sound foundation for local pricing, provided the pipeline treats every source as capable of failing quietly. The five defenses described here, three state cells, body sniffing, growth-based index comparison, staleness detection, and ordered fallbacks, share one design rule: each condition becomes data, and that data stays attached to the value until a person can see it. The cost model, its tests, and a sample rate file are published as the open-source cost model under the MIT license.
The Cookie That Never Came Back The first call authenticated the user. The second failed. I was integrating with a customer-provided SOAP service exposed through a single WSDL, with JAX-WS client classes generated by IBM Rational Application Developer from the WSDL and its XSDs. The first call authenticated the user and came back with a Set-Cookie header. Every call after it needed that cookie. The generated client sent none of it. Each invocation behaved as an independent conversation, and the workflow broke on its second step. No exception, no warning. Just a server that had forgotten who you are. The Built-in Answer, and Why It Failed The JAX-WS specification anticipated this: every proxy generated from a WSDL implements BindingProvider, and the spec defines a request-context property for exactly this purpose: Java CheckPort port = new Check().getCheckPort(); ((BindingProvider) port).getRequestContext() .put(BindingProvider.SESSION_MAINTAIN_PROPERTY, Boolean.TRUE); Set that flag, and the runtime is supposed to maintain session state. In most tutorials, this is where the article ends. Mine broke in deployment for three reasons. First is scope. The property is set on one proxy instance, and the runtime's cookie store lives with that proxy. RAD-generated code encourages pulling a fresh port per unit of work. Every fresh port gets a fresh store, so the authentication session is discarded when the method returns. Holding one proxy for the workflow fixes it, but pins the session lifetime to the object lifetime. Second is opacity. The spec does not say where the runtime keeps the cookie. The JDK's bundled implementation delegates to java.net.CookieHandler.getDefault(), null unless somebody installed one. IBM's runtime, which RAD-generated clients typically run on under WebSphere, does it its own way. Same client class, different behavior per container, and the failure is silent. Third is the obvious fix: install a default CookieManager inline. `CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL));` A JVM-wide static. Fine in a standalone client, but inside an application server it is shared by every deployed application and every outbound HTTP client in the JVM. It is also the wrong model for a server-side client: a CookieManager is built for a browser, where one cookie jar belongs to one user. A server handling concurrent requests would pool every user's session cookies into one store, and CookieManager applies RFC domain and path matching, so if the service sits behind a load balancer, the cookie is quietly dropped. The Options on the Table I wanted cookie handling scoped to the call, visible in the code, and independent of the container. The alternatives fell short. Holding one proxy for the whole workflow is the simplest change, but it marries the session to an object's lifetime, and the strain shows across components or threads. Installing a default CookieManager is one line and works in a standalone client, but inside an app server it becomes shared mutable state across applications and users, which rules it out for anything concurrent. Hand-rolling the client on plain HTTP gives full header control with no container surprises, but you lose the WSDL tooling: generated types, binding, error handling. Too much machinery to rebuild for several operations. A custom SOAP handler was the fourth option: roughly sixty lines you own, sitting in the message path of the existing generated client. Nothing JVM-wide, nothing container-specific, nothing hand-edited in generated classes. The costs: you match the cookie by name instead of trusting RFC rules, and you own the stored value's lifecycle. Acceptable for this integration. How the Handler Works The contract is SOAPHandler<SOAPMessageContext>; the direction comes from a boolean on the context. The listing below is illustrative, adapted from the class I used, and not presented as tested code. Java public boolean handleMessage(SOAPMessageContext ctx) { if (Boolean.TRUE.equals(ctx.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY))) { attachCookie(ctx); } else { captureCookie(ctx); } return true; } @SuppressWarnings("unchecked") private void captureCookie(SOAPMessageContext ctx) { Map<String, List<String>> headers = (Map<String, List<String>>) ctx.get(MessageContext.HTTP_RESPONSE_HEADERS); if (headers == null) { return; } for (Map.Entry<String, List<String>> entry : headers.entrySet()) { if (!"Set-Cookie".equalsIgnoreCase(entry.getKey())) { continue; } for (String raw : entry.getValue()) { String pair = raw.split(";", 2)[0].trim(); if (pair.startsWith("JSESSIONID=")) { SESSION_COOKIE.set(pair); } } } } @SuppressWarnings("unchecked") private void attachCookie(SOAPMessageContext ctx) { String cookie = SESSION_COOKIE.get(); if (cookie == null) { return; } Map<String, List<String>> existing = (Map<String, List<String>>) ctx.get(MessageContext.HTTP_REQUEST_HEADERS); Map<String, List<String>> headers = (existing == null) ? new HashMap<>() : new HashMap<>(existing); headers.put("Cookie", Collections.singletonList(cookie)); ctx.put(MessageContext.HTTP_REQUEST_HEADERS, headers); } public boolean handleFault(SOAPMessageContext ctx) { return true; } public void close(MessageContext ctx) { } public Set<QName> getHeaders() { return Collections.emptySet(); } } Three details in that listing exist because earlier versions broke. Copying the existing request headers before adding Cookie keeps the handler additive: putting a fresh map into HTTP_REQUEST_HEADERS replaces whatever was already there, and my first version did exactly that, until another handler down the chain set a custom authorization header that then vanished. Matching the header name case-insensitively is for portability, since a case-insensitive map is an implementation choice, not a spec guarantee. Splitting on the first semicolon keeps response attributes such as Path, Expires, and HttpOnly out of the request; echoing them back produces a Cookie value the server will not parse. Iterating over every value rather than taking index zero covers responses that set more than one cookie, where the session cookie is not guaranteed to be first. Why Threadlocal, and Where It Bites The integration ran several workflows concurrently, so a static field would leak one user's session into another's request. A ThreadLocal gives each thread its own value, which fits a workflow that runs start to finish on one thread. The cost is that thread pools reuse threads. If a workflow ends without clearing the value, the next request on that thread inherits a stale session cookie, and in the worst case sends one user's session identifier on another user's behalf. Clearing it when the workflow ends closes that hole: SESSION_COOKIE.remove(); Note that close() is intentionally empty: the container calls it at the end of each message exchange, not the workflow, so clearing there would discard the cookie from the authentication call. Cleanup belongs at the logout step or in a finally block around the workflow. And if the workflow hands off between threads, ThreadLocal is the wrong container: the cookie then belongs in whatever object already represents the conversation. Registering the handler JAX-WS lets handlers be declared in an external XML file: XML <?xml version="1.0" encoding="UTF-8"?> <handler-chains xmlns="http://java.sun.com/xml/ns/javaee"> <handler-chain> <handler> <handler-class>com.company.cc.handler.SessionCookieHandler</handler-class> </handler> </handler-chain> </handler-chains> The file sits in the same package as the generated service class, since the annotation resolves relative to that class: Java @WebServiceClient( name = "check", targetNamespace = "http://ws.example.com/", wsdlLocation = "META-INF/wsdl/spellingCheck.wsdl" ) @HandlerChain(file = "sessionCookieHandler.xml") public class Check extends Service { } Every proxy from that service now runs through the handler. One caveat: regenerating from an updated WSDL overwrites the service class and removes the annotation, so keep it in a small subclass or script the regeneration step. Edge Cases Cookie rotation needs no special handling: capture runs on every inbound response, so a new cookie mid-workflow replaces the old value on the next exchange. Expiry is different: a ThreadLocal cannot know when the server-side session times out, so an idle workflow keeps attaching a dead cookie. That fix belongs at the workflow level: catch the session-expired response and re-authenticate. Testing It without a Container A test can build a stub message context, inject HTTP_RESPONSE_HEADERS, run handleMessage, and check the Cookie header on the next outbound call. Cover the cases that burned earlier versions: two Set-Cookie values in one response, a header name in unexpected case, a pre-existing request map that must survive the copy, and the remove() at the end of the workflow. For integration confidence, run the real client against a stub endpoint that returns Set-Cookie on the first call and asserts the Cookie header on the second. The Takeaway JAX-WS has a session mechanism. It is scoped to a proxy instance, implemented differently across runtimes, and backed by a JVM-wide singleton that does not fit a server-side client handling concurrent users. A custom SOAP handler replaces it with a small class you own, an explicit scope you choose, and behavior identical wherever the application deploys. For teams maintaining legacy SOAP integrations that will not be modernized soon, that trade is usually the right one.
Every app that accepts photo uploads meets HEIC sooner or later. iPhones save in it by default. Neither Java nor Rust can read it out of the box, and the usual answer, libheif, is a C library: native packages in every container image, a C toolchain in the build, and a parser written in C reading files that strangers upload. I had the problem twice, in a Java product and in a Rust one. So I went the other way: decoders written in the language itself, built and improved by AI coding agents, with me steering. Here is how it went. 1. Make It Work, In Each Project One agent per codebase, each starting from an existing open-source decoder in its own language: Openize.HEIC for Java, HEIC decoder for Rust. The first versions lived inside each app. They read the photos, and that was enough to ship. They were ready in no time, given there were upstream libraries to start from, even if some features had to be added to match libheif. It was tempting to stop there. Nobody complains about an upload that works. 2. Extract and Iterate Next, I had each decoder pulled out as a standalone library, open source, with its own tests and releases, aligned with the best practices for each language (and mine). Then I asked each agent to iterate on its own: measure, improve, measure again. That alone went a long way. The Java decoder went from 19 seconds to 3.2 for our ten test photos. 3. Compare Everything With Everything Working alone, each agent eventually ran out of obvious ideas. What changed the game was putting both decoders next to libheif on the same machine with the same photos, and then comparing them stage by stage. Every gap became a lead. If another implementation did a step fast, there was a way to do it even faster, and the size of the gap told me whether it was worth chasing. A gap is a hint with a number attached. The comparison also kept us honest. It showed where we were ahead, and it showed that part of libheif's lead came from doing less work: it copied color samples where our decoder interpolated them. 4. Make the Agents Talk Here is the part that led to further improvements. Each agent, working in its own session, was told to talk with the other agent. I had the Rust agent brief the Java agent on what it had found, and the other way around. A gap one side had already closed became a recipe for the other. The Java decoder took the Rust decoder's entropy decoding approach; the Rust decoder took the Java decoder's parallel tiles and transform tricks. Later, the two agents, who were then aware of each other, even coordinated on their own, communicating which CPU cores each could benchmark on. Connecting them was the most useful step in the project. 5. Repeat On the Gaps That Remain Then we ran the same loop again: compare, find the largest remaining gap, close it, then measure. One more round on the Rust side closed the last single-core gap with libheif. The Java side tried the JVM's vector instructions and gained another 9 to 14%. Not every idea survived. The trick the C library is famous for, wide vector instructions, gained almost nothing in Rust: its slow parts were elsewhere. We measured it, wrote it down, and dropped it. Be Strict About Evidence A computer is never idle. Browsers, chat apps, other agents: everything competes for the CPU, and the numbers move with it. When we measured the very same code twice, back to back, the second run came out about 0.4% slower. Noise with a direction looks exactly like progress, or like a regression. So I asked for proof, not impressions: count CPU work instead of wall-clock time, run old and new in alternating order, split results by type of photo, and drop any "gain" smaller than the noise. Several changes didn't make it. One that looked good overall turned out to slow down rotated photos, which are most of what phones upload, until the agent found why. Where They Stand Today Measured together on the same machine, the same ten photos, all three doing the same work: 8 cores: Rust 0.46 s, Java 0.55 s, libheif 0.81 s1 core: Rust 1.38 s, libheif 1.45 s, Java 1.96 sRust can also interpolate color for sharper edges and still matches libheif: 1.45 s Memory is on par with libheif or below, and a long way from where the two decoders started. What I Take Away Iterating alone gets you far. Comparing gets you further. The gap between two implementations is the best to-do list.Cross-pollinating agents helps a lot. The ideas were there, in two separate sessions. Making them talk to each other triggered more optimizations.Be strict about evidence. Ask for measurements that hold up against noise, and be ready to throw away a change that doesn't.Two implementations in two languages beat one. Each made the other faster. Getting a third as reference (libheif) helped even more.Java is fast, too. “Java is slow” is a meme, not a measurement. Optimized the right way, it comes close to the C library on one core and beats it on eight. Both decoders are open source: Rust: github.com/Valuya/heic-decoder-rustJava: github.com/Valuya/openize-heic-java
Authentication Isn’t a Feature, It’s Frontend Architecture Whereas the authentication process can typically be implemented as an afterthought for any development team through just creating a login interface that will be connected to some sort of identity management solution, it will not be as straightforward when there is a need to develop a fully functional enterprise solution with the use of Angular and Spring Boot. From implementing single sign-on and multi-factor authentication in such cases with the use of such a front-end/back-end combination, it became evident that authentication is more of a constraint than a feature. Why SSO Changes More Than the Login Screen However, the idea of single sign-on appears to be fairly straightforward – you visit the identity provider and get your token, then that’s all. But that implies that the application will have to start working differently from within. From now on, the routes should have a guard to check if the user is authorized before any component gets initialized. Deep links have to be maintained during the redirect and return from the identity provider, which essentially means that the app needs to maintain the intended destination state in a process it does not control. Plus, since the token comes from an external provider on their own schedule, it needs to be treated by the frontend as an ordinary and expected situation where a token expires. In our platform, we implemented route guards checking token validity before resolving a protected route and token refreshing in the background mode without any interruptions to the user’s workflow. This part was more important for the whole implementation than the single sign-on flow itself, since a technically correct single sign-on flow that constantly interrupts users is still considered broken. The refresh mechanism brought its own nuances. In case multiple requests failed due to token expiration, an oblivious interceptor would spawn multiple refresh actions, while the majority of identity providers will revoke all previous refresh tokens upon the generation of a new one. It was resolved by having a request queue, where the first error will cause a single refresh action to be fired, while the rest of the pending requests would wait for this result and get re-fired with a fresh token. Should the refresh action fail, the queue would be cleared, and the user would go through the entire authentication process with his destination saved. Where Multi-Factor Authentication Adds Real Complexity One of the features MFA brings to the table is an additional level of state that the frontend has to handle properly: users might be authenticated but not yet fully verified. Confusing these two states is what makes most MFA setups break: they allow access to routes only available to fully authenticated users or require users to go through a verification step that they have already done in the current session. We considered the authentication state to be a state machine rather than a mere boolean value due to the existence of different states like the state of unauthenticated, the state of authenticated and waiting for verification, and the state of authenticated. Different permissions were applied in each of those states, and different transitions were possible between the different states. It allowed us to easily predict the results of the authentication code analysis and especially testing since we could test each transition between the different states. Step-up verification introduced yet another complication. Certain activities, like processing an important transaction, needed more recent verification than merely logging in. We simulated this by modeling it as a level of verification tied to a period of time within the session, thus allowing the back-end to request re-verification during certain important tasks while retaining the user’s work at the front-end. The Spring Boot Side of the Contract All of that required the backend to implement the exact same model that the frontend expected. The validation of tokens was done by our Spring Boot services on every single request instead of relying only on the frontend, and we were sending different and clearly recognizable error messages for token expiry, invalid tokens, and insufficient verification level instead of simply sending 401 all the time. And these differences are extremely important on the frontend – token expiry causes silent refreshing of tokens, invalid tokens cause the full authentication process, and insufficient verification level causes the user to be forced to pass through the MFA challenge once again, and not start the whole authentication from scratch. It took us way more coordination and planning to get this contract right than either side expected initially. The backend team could have considered all types of authentication errors to be the same thing and the frontend team could have thought that the error codes would translate perfectly into frontend states without any further checks. We recognized this problem at the very early stages mainly thanks to our SDET testing this boundary explicitly. What This Means for Compliance-Heavy Environments As the banking system is a highly regulated one, authentication was not only about the quality of the user experience; it was also an audit problem. With every state transition done, we needed to be able to explain and justify our choice of actions rather than just copying what worked before. This way we had to move toward using the explicit state machine rather than implicit, since the latter was difficult to prove. The Takeaway It is the choices about the design of the authentication procedure, the model of the state, semantic error management in the frontend-backend communication, token refreshing, and many more that will define the level of opposition that our team needs to overcome in order to make the app survive. The fact that our approach to the design of the authentication procedure as part of the frontend architecture from the very beginning was the reason for our success.
Production code gets scrutiny. Builds are watched, libraries are checked for known vulnerabilities, and someone is responsible when a service fails at night. The job that pulls in outside data usually escapes all of this. It began as a quick script, got scheduled, and has run untouched ever since. It has no tests, its only error handling is a catch-all exception block, and its author may have left the team long ago. The failures are rarely loud. A scraper does not crash when a site stops cooperating. It keeps running and keeps writing rows, and the rows quietly get worse. This article covers where pipelines actually break, how to respond to each kind of failure, and how to decide between maintaining the collection layer yourself and licensing it. The Block Page Is the Last Signal A CAPTCHA is the most visible sign of trouble, but sites test requests long before showing one. Several quieter checks decide whether the client looks like a real browser: TLS fingerprinting. Methods such as JA3 and the newer JA4 profile a client from its handshake. A stock Python or Node HTTP library is recognizable before any header is read. Changing the User-Agent string does nothing here, because the handshake gives the client away first.Rate limiting. Limits can attach to an IP, a session, or a client fingerprint. Some sites return a clean 429 with a Retry-After header. Many return a 403, a CAPTCHA, or an ordinary page with the data removed.Interstitials. Consent prompts and “checking your browser” pages often return status 200, as do regional redirects. A naive client logs a success without ever seeing the content.Honeypots. Links hidden with CSS are invisible to people. A crawler that follows every anchor walks straight into them, and the site can flag the session on the spot. Because these checks fail silently, the response code is the wrong thing to trust. Treat a 200 as a claim that needs verification. Compare each response against what that source normally returns. A page that usually weighs 150 KB and suddenly weighs 3 KB deserves a flag, whatever the status says. So does any page containing phrases typical of challenge screens, such as “verify you are human” or “checking your browser.” Both checks are cheap, and together they catch most soft blocks. Thresholds and phrase lists need tuning per target, since every site words its challenges differently. Even a pipeline that passes every check today will break when the target changes its markup. The checks have to keep running after launch. Different Failures Need Different Responses A scraper that reacts to every problem the same way will either hammer a site or give up too early. Each signal calls for its own handling: A 429 means slow down. Wait for the interval the server names in Retry-After, then resume.A persistent 403 means stop and investigate. More requests will not change the answer. The cause is usually a fingerprint, a policy, or an IP reputation problem that needs a human decision.A 200 with unexpected content means validate. Run the size and phrase checks, quarantine the response, and do not count it as a success.A 404 means record it and move on. A second attempt adds load and log noise without changing the outcome. For transient failures, retry with exponential backoff. Wait a second after the first failure, two after the second, four after the third, up to a sensible ceiling. Add random jitter to every wait. Without it, many workers retrying on the same schedule hit the site at the same moment and trigger the limit again. Rendering Adds a Second Problem More sites now deliver content through client-side JavaScript. A plain HTTP request returns an empty shell with a script tag and nothing to parse. Before reaching for a browser, open the network tab. Many pages load their data from a JSON endpoint that the client can call directly. That path is usually faster and cheaper than parsing rendered HTML, and it often changes less when the page design changes. It comes with a caveat. Undocumented endpoints carry no stability promise, so they can change or disappear without notice and need the same monitoring as any other target. Check the site’s terms and robots rules as well before depending on one. When no usable endpoint exists, a headless browser driven by Playwright or Puppeteer is the usual answer. It solves rendering and creates new costs: Each browser instance uses far more memory and CPU than an HTTP client, so throughput per machine drops sharply.Page loads take seconds instead of milliseconds, and flaky waits produce intermittent empty results.Detection systems look for automation signals such as navigator.webdriver, inconsistent canvas output, and unusual viewport settings. One habit prevents a large share of rendering bugs. Wait for a specific element that proves the content has loaded, not for a fixed number of seconds, and fail the job when that element never appears. A fixed sleep hides rendering failures. An explicit wait turns them into errors someone can see. Why the Pieces Are Harder Together Than Apart Proxy rotation, retry policy, and browser rendering are each manageable alone. Combined, they interact. A rotated IP changes the apparent location, which changes the consent banner, which changes the page structure your selector expects. A slow browser session can outlive the proxy connection it started on. Retries multiply browser launches and the cost that comes with them. Keeping all of this stable beyond a few requests a day is an engineering discipline of its own, separate from whatever the data is used for afterward. AI Workloads Raise the Bar A scraper built for a nightly batch load rarely suits AI use. A retrieval system needs a site’s structure mapped before it can decide what to embed. An agent that calls a scraper as a tool needs output it can parse immediately, returned fast enough to fit inside one tool call instead of a background job. Retrofitting a batch scraper for interactive use tends to expose every shortcut taken in the original build. Output format becomes a real requirement for these consumers. Clean Markdown or structured JSON saves a cleanup step that otherwise sits between collection and an embedding pipeline. Raw HTML still has a place when an application needs the full page structure. No format removes the need to validate content. Agents also introduce a security problem. When a model or an outside user can supply the URL that a scraper fetches, the scraper becomes a way to reach internal systems. This is server-side request forgery. A request aimed at a private address or a cloud metadata endpoint can expose data that was never meant to leave the network. Production deployments should restrict which destinations are allowed, resolve each hostname and check the resulting address before connecting, and apply the same checks to every redirect. Build or License Building means running several parts at once: Rotating proxy pool with enough geographic spread for the targets you needRetry queue with per-domain rate limitsBrowser cluster for JavaScript-heavy sitesParsing layer that changes whenever a target doesMonitoring that tells you when any of the above has quietly stopped working Each part is well documented. The difficulty is keeping them reliable together, which looks less like writing a script and more like operating a small production service with its own on-call rotation. Proxy coverage is the piece teams most often underestimate. Reaching residential and mobile addresses across many regions is a procurement and operations problem, not a coding one. For lighter needs, building may be more than the job requires. An official API, a published feed, or a scheduled HTTP client is often enough, and none of them needs this much machinery. Licensing moves some of that work to a provider. The category has moved well beyond proxy resale. Some managed services combine scraping, search, site mapping, and crawling under one API key. For example, a web data API can return HTML or Markdown, with structured formats such as JSON and CSV available for supported targets. This can reduce conversion work before the data reaches a database, a BI tool, or an embedding pipeline. Mapping and crawling endpoints also help with scoping. A map request can list a large share of a domain’s URLs up front, which lets a team estimate size and pick what deserves a full crawl. A crawl job can then collect thousands of pages in one pass, enough to cover a typical documentation site or product catalog. Licensing adds its own obligations. Check how the provider reports failures, whether you can see the raw response, how pricing changes at volume, where its IPs are located, and how much control you have over sessions and output. A managed service also does not take over responsibility for data quality, security, or compliance. When collecting personal data, consider why it is needed and whether its use is consistent with the original purpose of collection. Your application still has to detect unusable responses and decide what is permitted to collect. An opaque API that fails silently is the same problem you started with, moved to someone else’s servers. Plan for Recovery Whichever route you choose, assume jobs will fail midway. Failed work should produce diagnostics that name the target, the failure type, and whether it looks temporary. Collection should be restartable for the affected pages only, without repeating everything that already succeeded. Engineers also need to tell network errors from access restrictions, rendering errors and extraction errors. A dashboard of successful HTTP responses cannot do that when the records inside those responses are incomplete. Decide With a Pilot Which route is right depends on whether data collection is the product or only an input to it. If scraping is core to what you sell, owning the stack gives you control where it counts. If it feeds something else, every hour spent on proxies and fingerprints is an hour not spent on the actual product. The quickest way to find out is to run one representative target through both approaches at small scale for a few weeks. Track these for each: Engineering hours spent building and fixingShare of requests that produce usable data, measured by content and not by status codesCost per thousand usable pagesLatency per requestTime to recover after the target changes The number that matters is not how many requests the system completes. It is how reliably the system delivers usable data at a cost the team can sustain. A pilot on a production target settles that better than any vendor claim or internal estimate.
Agile
Career Development
Methodologies
Team Management
November 13, 2026
by Stefan Wolpers
CORE
Treating Web Scrapers Like Production Services
November 11, 2026 by Nitish Mane
AI-Assisted Refinement Preparation
November 10, 2026
by Stefan Wolpers
CORE
AI/ML
Big Data
Databases
IoT
November 13, 2026
by Stefan Wolpers
CORE
Designing for Hot Keys: When Distributed Systems Stop Distributing
November 13, 2026 by Jaswanth Mopathi
Cloud Architecture
Integration
Microservices
Performance
November 13, 2026 by Himanshu Mandhyan
Designing for Hot Keys: When Distributed Systems Stop Distributing
November 13, 2026 by Jaswanth Mopathi
Frameworks
Java
JavaScript
Languages
Tools
November 13, 2026 by Himanshu Mandhyan
November 13, 2026 by Dhana PAMULA
Deployment
DevOps and CI/CD
Maintenance
Monitoring and Observability
November 13, 2026 by Himanshu Mandhyan
Key ML Classification Metrics You Need to Understand
November 13, 2026 by Rakshath Naik
AI/ML
Java
JavaScript
Open Source
November 13, 2026
by Stefan Wolpers
CORE
From Database Changes to Real-Time Actions at Scale: Oracle GoldenGate and Java
November 13, 2026 by Sushant Sakolkar
AI Didn't Replace Software Engineering; It Changed the Bottleneck
November 13, 2026 by Jackson Stakeman