Testing, Deployment, and Maintenance

There's a particular kind of silence that happens on a call when a regulator asks, "Can you walk me through exactly how this number was calculated?" and the person on the other end has to say, "Let me get back to you." I've been on that call. It is not a fun place to be, and it usually isn't because anyone did anything wrong. It's because the pipeline that produced the number was never built to answer that question in the first place. Regulatory reporting, whether it's a liquidity return, a capital calculation, or something more niche, has a strange property compared to most data engineering work. Nobody outside the team really cares how the dashboard behind it looks. What they care about is whether you can prove, months or years later, exactly where every figure came from, who touched it, and why it changed between two reporting periods. That's a fundamentally different design goal than "make the report run fast," and honestly, a lot of pipelines get built for speed and convenience first, with auditability bolted on afterward as an afterthought. It shows. What "Audit-Ready" Actually Means in Practice I think people sometimes hear "audit-ready" and picture some heavyweight compliance layer sitting on top of a normal pipeline. In my experience, it's less about adding a layer and more about changing a handful of default behaviors early, before the pipeline exists in production. Four things tend to matter most. First, nothing gets overwritten. Ever. If a source record changes, you keep the old version and the new one, both timestamped. Second, every transformation step has to be reproducible from code, not from a person's memory of what they clicked in a tool three years ago. Third, there's a reconciliation point, a moment where the pipeline checks its own output against the source and either passes or stops. And fourth, a human actually looks at the number before it leaves the building. Not a dashboard glance, an actual documented sign-off with a name and a timestamp attached to it. Here's roughly how that fits together end to end. A couple of things in that picture are doing more work than they look like they are. The raw zone sitting right after ingestion, untouched and append-only, is what lets you go back eighteen months later and reconstruct exactly what the source system handed you on a given day, independent of whatever the curated tables look like now. I've genuinely had to do this, pulling raw records from a specific date to settle a dispute about whether a discrepancy came from us or from upstream. Without that raw layer, that conversation becomes a lot more uncomfortable. The Reconciliation Step Is Where Trust Actually Gets Built If I had to pick one part of this architecture that matters more than the rest, it's the reconciliation checkpoint. This is the moment where the pipeline stops behaving like a normal ETL job and starts behaving like a control. The logic itself isn't exotic. You compare row counts, sums, and control totals against the source, and you either match within tolerance or you don't. What matters is what happens on the "no" branch. A lot of pipelines I've inherited over the years treated a reconciliation break as a warning buried in a log file that nobody reads until someone complains. That's backward. A break should halt the pipeline and page someone, because a regulatory submission built on unreconciled data is worse than a late submission. I'd rather explain a delay than explain a restated figure. I'll admit there's a tension here worth naming honestly. Hard stops on every mismatch can grind a team down if your tolerance thresholds are too tight, and you end up with alert fatigue where people start ignoring pages because most of them turn out to be rounding noise from a currency conversion. Getting the thresholds right, informed by prior period baselines rather than guessed at, takes real tuning. It's not a solved problem you configure once and forget. Lineage and the Four-Eyes Check The governance band running underneath the pipeline, capturing lineage and version history, is what turns "I think this is right" into "here's the exact model, the exact commit, and the exact person who approved it." On Unity Catalog or an equivalent lineage tool, this is largely automatic once you've built your transformations in something like dbt rather than scattered SQL scripts. That's honestly one of the more underrated benefits of moving regulatory pipelines onto a tool with built-in lineage tracking. You stop having to build that tracking yourself. The human sign-off matters just as much, maybe more, even though it's the least technically interesting part of the whole design. A second person reviewing the output before submission, with their review logged, catches the kind of error that automated tests won't, like a number that's technically correct by every rule you wrote but obviously wrong to a person who understands the business context. A Small Piece of Advice If you're building one of these pipelines from scratch, resist the urge to treat the audit trail as something you'll add once the pipeline is "working." Retrofitting immutability and lineage onto tables that have already been overwritten a hundred times is painful, and in some cases you simply can't recover the history you needed. Build the raw, append-only layer first, even if it feels like overhead in week one. Six months later, when someone asks you to explain a number from last quarter, you'll be glad it's there.

A customer changes her email address in a mobile app on Friday afternoon. The app displays a green confirmation message: Your profile has been updated. The API response was 200 OK. The mobile team sees no error. The API dashboard counts the request as successful. On Monday, the customer receives an account statement at her old email address. The update never reached Salesforce. This is the kind of failure that frustrates everyone because nobody is obviously wrong. The mobile app did what the API told it to do. The API gateway recorded a successful response. The Mule application may even have logged the Salesforce timeout. But the business action did not happen. The customer did not care that the JSON payload was valid or that MuleSoft accepted the request. She cared that her email address changed. That is the gap many integration APIs miss: technical success is not the same as business completion. A 200 OK response can be completely valid at the HTTP level while being dangerously misleading at the business level. This article is about fixing that gap. The Real Problem: Green Dashboards, Failed Business Operations Here is the common sequence. The mobile app calls a customer-profile API. MuleSoft validates the request and invokes Salesforce. Salesforce is slow or temporarily unavailable. The Mule flow catches the error, logs it, and continues so that the client does not receive an ugly error message. The caller gets 200 OK. The update never happens. Figure 1 — The Green Dashboard Failure A support ticket usually appears much later: “The customer says she updated her email, but Salesforce still has the old one. Can someone check?” At that point, the support engineer has to search logs, identify the correlation ID, discover a suppressed connector error, and explain why the API told the customer the opposite of what happened. That is not a monitoring problem. It is an API-contract problem. How MuleSoft Can Accidentally Turn Failure into Success The risky part is often a well-intentioned error handler. A developer does not want to expose Salesforce exceptions, stack traces, or internal system names to a mobile application. That is the right instinct. But hiding technical details is different from hiding the fact that the operation failed. Consider this Mule error handler: XML <error-handler> <on-error-continue type="HTTP:CONNECTIVITY"> <logger level="ERROR" message="#['Salesforce update failed: ' ++ error.description]" /> <set-payload value='#[{ status: "SUCCESS", message: "Customer updated successfully" }]' /> </on-error-continue> </error-handler> The log captures the failure. The client receives success. The word that changes everything is continue. In Mule 4, on-error-continue treats the error as handled and allows the owning flow to complete. If the flow owns an HTTP request, that can result in a successful response even though a downstream call failed. MuleSoft’s own error-handling documentation shows this behavior. on-error-continue is not wrong. It is useful when the failed step is optional. For example: A customer order is created, but a noncritical analytics event fails.A product API succeeds, but an optional cache refresh fails.A notification preference update succeeds, but a marketing enrichment call fails. In those cases, continuing can be reasonable. It is not reasonable when the failed call is the business action the API promised to perform. If the API says “update customer email,” and Salesforce is the system of record for customer email, then a Salesforce failure means the update is not complete. The response must say so. Stop Asking “Did the API Work?” Ask a better question: “When this API returns success, what business fact can the caller rely on?” For a customer-email update, the answer should be specific: “The customer’s email address has been updated in the system of record.” If that cannot be guaranteed before the response is sent, the API has two honest choices: Return a failure because the requested action did not complete.Return 202 Accepted because the action was safely accepted for later processing. What it should not do is return 200 OK because the request reached MuleSoft. Figure 2 — Three Very Different Meanings of “Success” This is the key design decision for command APIs: API responseWhat it should meanExample200 OK or 201 CreatedThe requested business action completedSalesforce confirmed the email update202 AcceptedThe request was durably saved and will be processed laterThe update command was written to a queue or database400 Bad RequestThe caller must fix the requestInvalid email format or required field missing409 ConflictThe request conflicts with known stateSame idempotency key used with different payload503 Service UnavailableA required dependency prevented completionSalesforce is unavailable and no durable request was created A successful HTTP response should not be a polite way of saying, “We will hope for the best.” The Synchronous Contract: Return 200 Only After the System of Record Confirms For a small, immediate update such as a customer email change, a synchronous API can be appropriate. The call waits for Salesforce. If Salesforce confirms the update, return 200 OK. If Salesforce is unavailable, return a truthful temporary failure. Plain Text sequenceDiagram participant App as Mobile App participant Mule as MuleSoft API participant CRM as Salesforce App->>Mule: PUT /customers/4821/email Mule->>CRM: Update email address alt Salesforce confirms update CRM-->>Mule: 204 No Content Mule-->>App: 200 OK<br/>{status: COMPLETED} else Salesforce unavailable CRM--xMule: Timeout Mule-->>App: 503 Service Unavailable<br/>{status: NOT_COMPLETED} end Here is the Mule flow shape: XML <flow name="update-customer-email"> <http:listener config-ref="HTTP_Listener" path="/customers/{customerId}/email" allowedMethods="PUT"> <http:response statusCode="#[vars.httpStatus default 200]" /> </http:listener> <try> <http:request config-ref="Salesforce_Request" method="PATCH" path="/services/data/v61.0/sobjects/Contact/#{attributes.uriParams.customerId}"> <error-mapping sourceType="HTTP:CONNECTIVITY" targetType="CUSTOMER:CRM_UNAVAILABLE" /> <error-mapping sourceType="HTTP:TIMEOUT" targetType="CUSTOMER:CRM_UNAVAILABLE" /> </http:request> <set-variable variableName="httpStatus" value="200" /> <set-payload value='#[{ status: "COMPLETED", customerId: attributes.uriParams.customerId, correlationId: correlationId }]' /> <error-handler> <on-error-propagate type="CUSTOMER:CRM_UNAVAILABLE"> <set-variable variableName="httpStatus" value="503" /> <set-payload value='#[{ status: "NOT_COMPLETED", code: "CRM_UNAVAILABLE", message: "The customer update could not be completed. Retry later using the same idempotency key.", correlationId: correlationId }]' /> </on-error-propagate> </error-handler> </try> </flow> The exact XML varies by project, but the design should not: Translate connector errors into meaningful application errors.Do not expose raw Salesforce or Mule exceptions.Do not convert a required downstream failure into success.Include a correlation ID in every outcome.Tell callers whether retrying is appropriate. MuleSoft supports custom error mappings precisely so a connectivity failure from one dependency can be handled differently from a failure in another. See the Mule error model. A 503 Can Be a Better Customer Experience Than a Fake 200 At first, returning 503 Service Unavailable sounds harsh. But compare the two customer experiences. Fake success Plain Text Friday: “Your email was updated.” Monday: Sensitive statement sent to old email. Tuesday: Customer contacts support. Truthful temporary failure Plain Text Friday: “We could not update your email right now. Please try again.” Friday, 30 seconds later: Customer retries. Friday: Update succeeds and is confirmed. A truthful failure is inconvenient. A false success is expensive. For retryable failures, return a clear response: HTTP HTTP/1.1 503 Service Unavailable Retry-After: 30 Idempotency-Key: 7a88e9dc-e1bd-4f38-a47b-0fc009392c78 X-Correlation-ID: 49d7d0a2-e771-43b7-ae4d-a9055ab21f3d { "status": "NOT_COMPLETED", "code": "CRM_UNAVAILABLE", "message": "The customer update could not be completed. Retry after 30 seconds using the same idempotency key.", "correlationId": "49d7d0a2-e771-43b7-ae4d-a9055ab21f3d" } This response gives the client application something useful: A reason category, without leaking internal implementation detailsA retry intervalA stable idempotency key for safe retryA correlation ID for support investigation That is far more useful than a green checkmark built on a hidden error. When 202 Accepted Is the Better Design Not every operation should wait for the downstream system before responding. Imagine a new employee onboarding request: Plain Text HR system → MuleSoft → Identity provider → Active Directory → Laptop workflow → Payroll This is not a two-second API call. It is a business process. Returning 200 OK after the HR request reaches MuleSoft would be misleading. But returning 503 because Active Directory takes time would also be wrong. The honest model is asynchronous. The API stores the request durably, assigns a tracking ID, and returns 202 Accepted. HTTP HTTP/1.1 202 Accepted Location: /employee-provisioning-requests/a0b6ef54-91d6-4a3c-ae76-e2d326f73ad5 { "requestId": "a0b6ef54-91d6-4a3c-ae76-e2d326f73ad5", "status": "PENDING", "correlationId": "49d7d0a2-e771-43b7-ae4d-a9055ab21f3d" } The client can then retrieve the real outcome: HTTP GET /employee-provisioning-requests/a0b6ef54-91d6-4a3c-ae76-e2d326f73ad5 HTTP { "requestId": "a0b6ef54-91d6-4a3c-ae76-e2d326f73ad5", "status": "COMPLETED", "completedAt": "2026-08-29T16:45:12Z" } Or, if something fails: JSON { "requestId": "a0b6ef54-91d6-4a3c-ae76-e2d326f73ad5", "status": "FAILED", "code": "IDENTITY_VALIDATION_ERROR", "message": "The employee username conflicts with an existing account." } Figure 3 — The Honest Asynchronous Contract The critical rule is simple: Return 202 Accepted only when the request has been safely stored somewhere durable. A request held only in memory is not accepted. A message that has not been committed to a queue is not accepted. A flow that merely intends to run later is not accepted. The API must be able to prove that it owns the request before it tells the caller to stop waiting. Measure Completion, Not Just API Availability A dashboard showing “99.99% of requests returned 2xx” can be dangerously comforting. For command APIs, measure three separate things: metricwhat it tells youHTTP response-code rateWhat the API told clientsDownstream completion rateWhether the target business system completed the actionTime to completionHow long customers wait for the real outcome For asynchronous work, also monitor: Pending requests by business operationAge of the oldest pending requestFailures by downstream systemRetry attempts and retry exhaustionRequests requiring manual interventionDuplicate requests prevented through idempotency A useful alert is not: “The API has 500 errors.” A useful alert is: “Customer updates accepted more than 10 minutes ago have not reached COMPLETED or FAILED.” That alert tells the operations team what is actually at risk. The Question to Ask in Every API Design Review Before approving an integration API, ask one question: “If this endpoint returns 200 OK, which system has definitely completed which business action?” If the answer is “MuleSoft received the request,” the contract is not finished. If the answer is “Salesforce confirmed the customer update,” then 200 OK is meaningful. If the answer is “the request was safely stored for later processing,” return 202 Accepted. If the answer is “we do not know because the dependency timed out,” return a truthful retryable failure—not success. That one question catches a surprising number of production defects before they become support tickets. Final Takeaway The goal is not to return more errors. The goal is to stop lying with success. A 200 OK should mean the business operation completed. A 202 Accepted should mean the operation was safely accepted for later completion. A 503 should mean the operation did not complete and can be retried safely. Anything else creates the worst kind of integration failure: one that is technically green, operationally invisible, and painfully real for the customer.

If you run Go services on Kubernetes, you know the ritual. You update a Secret or a ConfigMap, and then you do one of these: kubectl rollout restart deployment/my-app and eat the disruption.Install Reloader so it restarts your pods for you when a Secret changes.Mount the Secret as a volume and cross your fingers that your app re-reads the file (spoiler: most apps read it once at boot). All three share the same flaw: the way you "pick up" a config change is to throw away the process and start a new one. For a rotating database password, that means dropping every in-flight request, cold connection pools, and a latency spike — all to change one string. I built mamori partly because of this exact frustration. mamori watches your Kubernetes Secrets and ConfigMaps with native informers, and when a value changes it reconciles the new value into your running Go process — typed, validated, and atomically swapped — without restarting anything. New to mamori? Start with the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go, for the full picture. This post is the Kubernetes-focused deep cut. The Two Schemes You'll Use mamori's Kubernetes provider gives you two ref schemes: RefReads fromNotesk8s-secret://<namespace>/<name>#<key>a Secretvalue is base64-decoded for youk8s-cm://<namespace>/<name>#<key>a ConfigMapplain string values Drop the #key and you get the whole Secret/ConfigMap as a JSON map, which you can decode into a nested struct with flatten. A few concrete examples: Plain Text k8s-secret://prod/db-creds#password # one key from a Secret, base64-decoded k8s-secret://prod/tls#ca.crt # a certificate as []byte k8s-cm://prod/app-config#log_level # one key from a ConfigMap k8s-cm://prod/app-config # the whole ConfigMap as a JSON map Setup The provider lives in its own module. Note the import path is providers/k8s (not providers/kubernetes): Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/k8s Register it with a blank import — same convention as database/sql drivers. Its init() wires the k8s-secret and k8s-cm schemes into mamori: Go import _ "github.com/xavidop/mamori/providers/k8s" In-cluster, the provider uses the pod's mounted ServiceAccount automatically. The only thing you owe it is RBAC: it needs to get, list, and watch the resources you reference in the namespaces you reference. YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: mamori-reader namespace: prod rules: - apiGroups: [""] resources: ["secrets", "configmaps"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: mamori-reader namespace: prod subjects: - kind: ServiceAccount name: my-app namespace: prod roleRef: kind: Role name: mamori-reader apiGroup: rbac.authorization.k8s.io The watch verb is the important one — it's what lets mamori use an informer instead of hammering the API server with polls. Load Once, Then Watch Forever Describe what you need as a struct. mamori resolves every ref, applies defaults, and validates — all before your main gets a config: Go package main import ( "context" "log" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/k8s" ) type Config struct { DBPassword secret.String `source:"k8s-secret://prod/db-creds#password"` LogLevel string `source:"k8s-cm://prod/app-config#log_level" default:"info" validate:"oneof=debug info warn error"` MaxConns int `source:"k8s-cm://prod/app-config#max_conns" default:"10" validate:"gte=1,lte=500"` CACert []byte `source:"k8s-secret://prod/tls#ca.crt"` } func main() { cfg, err := mamori.Load[Config](context.Background()) if err != nil { log.Fatal(err) } _ = cfg // fully resolved, defaulted, validated } Notice DBPassword is a secret.String, not a plain string. That type redacts itself in logs, fmt, and JSON — so a stray log.Printf("%+v", cfg) prints [REDACTED] instead of leaking the password. You only ever see the plaintext when you explicitly call .Reveal(). Now the part that matters. Swap Load for Watch, and the moment someone runs kubectl edit secret db-creds, your process reacts: Go w, err := mamori.Watch[Config](ctx, mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // Rebuild the pool with the rotated password — live, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) } for _, f := range ev.Fields { log.Printf("reconciled %s: %s -> %s", f.Path, f.OldVersion, f.NewVersion) } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // lock-free snapshot; always the last VALID config w.Get() is lock-free and always returns the last valid configuration. If someone pushes a ConfigMap with log_level: banana, the validate:"oneof=..." tag rejects it: the update is dropped atomically, Get() keeps returning the good value, and OnError receives a *ValidationError. A bad edit never takes your service into a broken state. Informers, Not Polling This is the difference between mamori and the mount-a-file-and-poll approach. The Kubernetes provider implements native watching backed by the client-go informer machinery, so change detection is push-based — the API server tells your process the instant a Secret changes, rather than your process asking every N seconds. Practically, that means: Near-instant reconciliation. No polling interval to tune, no min(sync, interval) latency floor like the kubelet's ~60s volume refresh.No API-server hammering. One watch connection per resource, shared, instead of a GET loop per pod.Graceful fallback. If a native watch fails to start for some reason, mamori automatically falls back to polling with jittered intervals, so you never lose reconciliation entirely. Compare that to the alternatives: ApproachHow a change is picked upRestart?kubectl rollout restartyou, manuallyyesReloader / Stakateroperator restarts the podyesSecret mounted as volumekubelet re-syncs file (~60s), if your app re-reads itno, but you must re-readmamoriinformer pushes change → validated atomic swap → callbackno Whole ConfigMaps and JSON Blobs Sometimes you keep a structured blob in one key. Decode it straight into a nested struct with flatten: Go type Config struct { Redis RedisConfig `source:"k8s-secret://prod/redis-conn" flatten:"json"` } type RedisConfig struct { Addr string `mapstructure:"addr"` Password secret.String `mapstructure:"password"` DB int `mapstructure:"db"` } That reads the whole redis-conn Secret (a JSON document), decodes it, and — because Password is a secret.String — keeps the password redacted throughout. A Note on TLS Certificates Certificates rotate too, and a cert reload is the perfect no-restart use case. Because file and Secret updates for certs usually want to apply immediately (no coalescing delay), you can set ?debounce=0 on the ref: Go type TLS struct { Cert []byte `source:"k8s-secret://prod/tls#tls.crt?debounce=0"` Key secret.Bytes `source:"k8s-secret://prod/tls#tls.key?debounce=0"` } Wire the OnChange callback to swap the *tls.Certificate your server hands out in its GetCertificate hook, and you've got zero-downtime cert rotation from a Kubernetes Secret. When You Don't Need This Be honest with yourself: if your config is a log level from a ConfigMap that only ever changes on deploy, you don't need runtime watching — a plain Load at boot is fine, and mamori is happy to do just that. And if you already mount a Secret as a file and genuinely re-read it, mamori's built-in file:// provider (with fsnotify) covers that too, without any Kubernetes RBAC at all. mamori earns its keep the moment a value changes while your process is running and you'd rather rotate a pool than recycle a pod. Wrapping Up Kubernetes already tells you the instant a Secret changes — via informers. mamori just connects that signal to a typed, validated, redacted struct inside your Go process, so a rotation becomes a callback instead of a rollout. Shell go get github.com/xavidop/mamori/providers/k8s Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and currently at v0.1.0, so issues and feedback are very welcome. Happy shipping!

By Xavier Portilla Edo CORE

A pod stuck in Pending is one of the most misleading states in Kubernetes. Nothing looks wrong. There's no error banner, no red text screaming at you. The scheduler just quietly gave up finding a home for it and is waiting for the world to change. I've stared at that status more times than I'd like to admit, usually at the worst possible hour, usually while someone in a chat window is asking why traffic isn't going through. Eleven minutes is about how long it took once, on a cluster that had close to 40% CPU sitting free the entire time. That gap between "there's capacity somewhere" and "this pod actually got scheduled" is what this article is really about. This is capacity spillover: demand outgrows what the current set of nodes can absorb, and the cluster needs somewhere for the overflow to land. Handle that well and it's a blip nobody notices. Handle it badly, and it's an outage that shows up in a customer complaint before it shows up in your dashboard. I'll be honest, a few years ago I would have called this an edge case worth a paragraph in the docs and moved on. Not anymore. Ask around, and you'll find most people running workloads in the cloud right now have a story about a specific instance type just not being there when they needed it, GPU-backed machines most visibly, but I've seen it happen with fairly ordinary compute SKUs too, usually right in the middle of a demand spike nobody scheduled around. That's really the whole case for spillover, once you strip away the architecture diagrams. Without it, what you get during a capacity crunch is close to random: whichever pod happens to hit the scheduler first grabs whatever's left, and everyone else waits and hopes. With it, the randomness mostly goes away. Workloads land where you decided they should, in the order you decided matters, instead of wherever the scheduler happened to find room first. New pod needs scheduling -> primary node pool checked -> if room is available, schedule normally; if the pool is full, spill to the secondary pool (overflow capacity). Why Scheduling Failures Happen in the First Place The scheduler's job sounds almost too simple: find a node with enough free CPU and memory for the pod's requests. In practice, a pod can fail to schedule for reasons that have nothing to do with the cluster actually running out of room. Here's the part that trips people up the first time they see it. A cluster can have plenty of free CPU in aggregate and still fail to place a memory-heavy pod, because no single node has enough memory available on its own. Resource requests get evaluated per node, not pooled across the whole cluster. I've had a coworker insist this was a bug the first time they saw it happen. It isn't. It's just how bin-packing works when nobody told you that's what you were looking at. Taints, tolerations, and node affinity narrow the field further. A pod that needs a GPU node, or needs to steer clear of a specific zone, only ever sees the slice of the cluster that matches. Spare capacity can be sitting right there, and the pod simply isn't allowed near it. And then there's the timing problem, which is the one that actually pages people. Even when the cluster genuinely needs more nodes and the autoscaler agrees, provisioning a node isn't instant. There's a real window between "we need capacity" and "capacity exists," and whatever's waiting in that window experiences it as a failure, not as a system doing exactly what it was designed to do. What Spillover Actually Looks Like "Spillover" gets used loosely, but it covers a few genuinely different situations, and I think that's where a lot of confusion comes from. People nod along in a meeting assuming everyone means the same thing, and they don't. The first one, and honestly the one most teams reach for first, is spilling to a different node pool entirely. You run a primary pool sized for steady-state load, and a secondary pool, usually spot or preemptible, that only spins up once the primary's full. Interruption-tolerant stuff gets pushed there when things get tight, which keeps the expensive, stable pool free for whatever actually can't afford to get evicted. It's an easy sell to whoever owns the cloud bill, which is probably why it's the version everyone sets up first. Then there's spilling across zones, or in the worst cases, regions. A single zone running out of a specific instance type during a demand spike isn't some rare event you plan a tabletop exercise around; it's just a Tuesday at this point. So you spread the workload wider. Fine in theory. I've watched a "let's just spread it across zones" fix get shipped on a Friday and turn into a cross-region latency ticket by the following Wednesday, because nobody had checked whether the workload actually tolerated that. And there's a third flavor that's less about running out of room and more about not wanting to spend the money. Technically, there's room on the expensive reserved instances. The better default is letting things spill onto cheap infrastructure first, and only reaching for the premium stuff when nothing cheaper is left over. Build it the other way, and you won't find out until the next billing cycle, which is a genuinely bad time to find out. Designing for It Instead of Reacting to It The cluster autoscaler, or something like Karpenter, handles the mechanics of adding nodes once pods can't be placed anywhere. That's useful, but it's also reactive by definition, closer to a fire alarm than a smoke detector. By the time it fires, everyone already knows something's wrong. A few things actually close the gap between "capacity ran out" and "someone got paged," in my experience. Pod priority and preemption are the ones people underestimate. Not every workload deserves an equal claim on scarce capacity, and treating them as if they do just means everything fails to schedule at once, in whatever order the scheduler happened to try them, which helps nobody. Give the stuff that actually matters a higher PriorityClass and let it evict lower-priority pods when things get tight. Over-provisioning on purpose sounds wasteful until you sit with it for a second. A handful of low-priority placeholder pods, doing nothing except holding a spot, get evicted the instant real demand shows up, and the real workload schedules immediately instead of waiting on a node to boot. Small constant cost, much faster response when it actually counts. Provisioning speed is the one that surprised me. Tools that can add nodes directly, without needing to fit some pre-defined pool shape first, shrink that window between "pod can't schedule" and "a node exists to put it on" a lot more than I expected going in. That window is basically where the pager lives. And testing failure on purpose, which is the step almost everyone skips. It's easy to assume spillover works because it's configured correctly on paper. Deliberately draining a node pool in a test environment and watching whether pods actually land where they're supposed to, in the order they're supposed to, finds a category of misconfiguration you will never catch by reading YAML, however carefully you read it. The Part That's Easy to Skip Past Spillover design tends to get treated as a purely technical exercise. Pick the autoscaler, define the pools, set the priorities, done. The harder question usually isn't technical at all: who decides which workloads are allowed to spill onto cheaper or less reliable capacity, and who's on the hook when a decision made for cost reasons turns into someone else's latency problem three sprints later. Getting the mechanics right is necessary. It's not enough on its own, because nobody's actually agreed on the policy those mechanics are supposed to enforce, and that conversation almost always happens after the first bad incident instead of before it. It's happened to every team I've worked with in some form. I'd rather have that conversation early, even if it's a little awkward and nobody wants to own the answer yet. It beats the alternative meeting, the one scheduled at 9 am the morning after something broke.

If you need to generate prerequisite test data, such as adding dummy records through an API, before executing your test suite, using Global Setup in Playwright is a clean, maintainable, and scalable approach. It ensures all required data is prepared upfront, keeping your test execution structured and reliable. What Is Global Setup? The Global Setup in Playwright runs once before the entire test suite begins execution. It is typically used for activities such as creating test data, seeding the database, generating authentication tokens, preparing the test environment, or cleaning up old data. Since it executes only once at the start of the run, it helps ensure efficient and consistent test initialization across the suite. How to Create a Global Setup in a Playwright TypeScript Project The following steps need to be followed to create a globalSetup in the Playwright project: Step 1: Create a global-setup.ts file in the Root Folder of the Project TypeScript import orders from './test_data/orders.json' with {type: 'json'}; async function globalSetup() { console.log('Running Global Setup...'); const apiContext = await request.newContext(); const response = await apiContext.post("http://localhost:3004/addOrder/", { data: orders, }); if(response.status() !=201) { throw new Error ("Failed to create setup test data!"); } console.log('Test data created successfully'); } export default globalSetup; The globalSetup() function will insert test data into the RESTful e-commerce demo application. The test data for creating new orders is supplied using the following orders.json file. JSON [ { "user_id": "1", "product_id": "79", "product_name": "5 star 10gm Chocobar", "product_amount": 5, "qty": 1, "tax_amt": 0.5, "total_amt": 5.5 } ] The data is required to validate the GET API tests, which will execute immediately after the Playwright test command is triggered. Step 2: Register globalSetup in playwright.config.ts TypeScript import { defineConfig } from '@playwright/test'; export default defineConfig({ globalSetup: './global-setup', }); The globalSetup property tells Playwright to execute the file once before the entire test suite runs. This setup file is typically used to perform pre-test activities such as creating test data or preparing the test environment, ensuring that all tests start with the required initial state. tsconfig.json: JSON { "compilerOptions": { "module": "NodeNext", "moduleResolution": "NodeNext", "resolveJsonModule": true, "types": ["node", "@playwright/test"] } } In this case, “require.resolve()” is not used because the project is configured to use ES Modules ("module": "NodeNext" and "moduleResolution": "NodeNext" in tsconfig.json). Since the project runs in ESM mode, Playwright can directly resolve the setup file using a string path ('./global-setup') instead of relying on CommonJS-based require.resolve(). Step 3: Run the Test The tests are already in place; only global-setup.ts needs to run to feed data into the system before running them. The script for running the tests is updated in the package.json file: JSON "scripts": { "test": "npx playwright test" }, Let’s run the tests by executing the following command in the terminal: Plain Text npm run test The following is a screenshot of the output after running the tests: It can be seen that the test data was created using the global setup before running the tests. How This Helps in Your API Testing Instead of creating the order in the first test and relying on test execution order, we can use a more robust approach by setting up the required data beforehand. By creating test data once during global setup, the tests remain independent, reducing flaky dependencies and unintended failures. This approach aligns with Playwright best practices and helps build a stable, maintainable automation test suite. Important Points to Note About the Global Setup It runs once per test run.It does NOT share variables directly with tests.Use files or environment variables to pass data.Assertions should not be put in the global setup, except for setup validation. Check out how to verify response data with Playwright TypeScript for a detailed understaning on assertions. Summary Global setup in Playwright runs once before the entire test suite and is primarily used for preparing the test environment before running the tests. It helps keep tests independent and avoids relying on execution order. However, since it runs only once and does not directly share variables with tests, any required data must be passed through files, environment variables, or external storage mechanisms. Happy testing!

By Faisal Khatri CORE

API testing is the practice of verifying that an API behaves as expected. In Agile environments, teams often perform it early in the SDLC as part of shift-left testing, enabling faster feedback and helping them identify and fix defects early. Playwright is a modern test automation framework that’s popular among testers and developers. Built by Microsoft, it makes it easy to automate both web and API testing across multiple programming languages, all within a single tool. In this tutorial, we will focus on testing APIs using Playwright TypeScript. By the end of this article, you should have a clear understanding of writing automation tests for PUT, PATCH, and DELETE API requests with Playwright TypeScript. Check out the previous articles to learn how to test GET API request and verify response data with Playwright TypeScript. What are PUT Requests? PUT requests are commonly used to update an existing resource by replacing its current data with the data provided in the request body. They are similar to POST requests; the Content-Type header plays an important role because it tells the server how to process the data being sent. When a PUT request is successful, it typically returns a 200 status along with the updated resource in the response. However, this can vary by API design; some APIs may return only a status code without a response body. How to Test PUT API Requests With Playwright TypeScript Playwright offers a request API that lets us create and manage HTTP request contexts easily. We’ll use the PUT /updateOrder API that updates the available order using its order ID. This API is a part of the RESTful e-commerce application available on GitHub. This API requires an authentication token to validate the request and authorize the update; if the token is missing or invalid, the request will fail. The order ID needs to be passed as a path parameter so the correct order can be identified, and the updated details should be included in the request body in JSON format. Since this is a PUT request, you must send the entire order object, not just the field being modified. Even for a small change, the full set of order data is required. We’ll use the following test scenario to demonstrate testing the PUT API request with Playwright TypeScript: Markdown ## Test Scenario Title: Update the existing orders in the system. ### Pre-condition: Valid orders are available in the system #### Test 1. Update all the order details of order_id "1" 2. Verify that the Status Code 200 is returned in the response 3. Assert that the order details have been updated correctly To implement this test scenario, we’ll need to perform the following steps: Hit the Auth API and extract the token.Write a test to send the PUT API request and pass the token extracted in step 1. Step 1: Create a Fixture to Generate and Extract the Token We’ll need the auth token for PUT, PATCH, and DELETE API requests, so let’s create a fixture that hits the POST /auth API and extracts the token for use across other tests. Let’s create a new fixture file auth.fixture.ts, in the fixtures folder in the project’s root directory. TypeScript import { test as base, expect } from "@playwright/test"; import dotenv from "dotenv"; dotenv.config(); type MyFixtures = { token: string; }; export const test = base.extend<MyFixtures>({ token: async ({ request }, use) => { const response = await request.post("http://localhost:3004/auth", { data: JSON.stringify({ username: process.env.USERNAME, password: process.env.PASSWORD, }), headers: { "Content-Type": "application/json", }, }); expect(response.status()).toBe(201); const responseBody = await response.json(); await use(responseBody.token); }, }); export { expect } from '@playwright/test'; A custom Playwright test setup is created by extending the base test object and adding a reusable token fixture. It uses the dotenv library to load environment variables like USERNAME and PASSWORD, ensuring sensitive data is not hardcoded, and defines a TypeScript type to make the token available with proper type safety. A .env file is created containing the credentials and placed in the project’s root folder. This ensures best practices are followed and sensitive information, such as usernames and passwords, is not exposed. As this is a demo API, the credentials used with the POST /auth endpoint are provided below. In a real-time project, the .env file should be added to the .gitignore file to ensure it is not committed to the remote repository. Plain Text #.env USERNAME=admin PASSWORD=secretPass123 Inside the fixture, an authentication API is called to generate a token; the response is validated, and the token is extracted and passed to the tests using the use() function. Finally, expect is re-exported so that both test and expect can be imported from this single fixture file, keeping test imports clean and consistent. Step 2: Updating the Order With a PUT Request and Verifying the Response Let’s create a new spec file to update the order using the PUT /updateOrder API request. TypeScript # 5_put-test.spec.ts import { test, expect } from "../fixtures/auth.fixture.js"; test("Update Order using PUT API Request", async ({ request, token }) => { const updatedOrder = { user_id: "89", product_id: "124", product_name: "Lindt Choclate", product_amount: 50, qty: 3, tax_amt: 10, total_amt: 160, }; const id = 1; const response = await request.put( `http://localhost:3004/updateOrder/${id}`, { data: JSON.stringify(updatedOrder), headers: { "Content-Type": "application/json", Authorization: `${token}`, }, }, ); const responseBody = await response.json(); expect(response.status()).toBe(200); expect(responseBody).toEqual( expect.objectContaining({ message: "Order updated successfully!", order: expect.objectContaining({ product_id: "124", product_name: "Lindt Choclate", }), }), ); }); Code Walkthrough Using the auth fixture for getting the token: The test imports a custom test from the auth fixture, which automatically provides the token as part of the test context. This token is generated before the test runs and injected via Playwright fixtures, eliminating the need to manually authenticate in each test.Request body: The updatedOrder object represents the payload sent in the request to update an existing order. It contains the required fields for the order update request body. This object is later serialized into JSON format before being sent to the API.Sending the PUT request: The request.put() function is used to send an HTTP PUT request to update the order with a specific ID. The request includes the serialized request body and headers such as Content-Type and Authorization, where the token is passed for authentication. This step performs the actual API interaction to update the order in the system.Dynamic endpoint handling: The order ID is stored in a variable and dynamically appended to the API endpoint using template literals. This makes the test more maintainable and reusable.Verifying the response: The response is first converted to JSON, and the status code is validated to be 200, indicating a successful update. Then, assertions are made using expect.ObjectContaining that partially validates the response body, ensuring that the success message and updated order details match expected values. Test Execution Before running the test, the RESTful e-commerce application should be started using Docker by running the following command from the terminal: Plain Text docker compose -f docker-compose-ecommerce.yml up -d The following command should be run from the terminal: Plain Text npx playwright test tests/5_put-test.spec.ts The following test output shows that the test was executed successfully: What Are PATCH Requests? A PATCH request is used when we want to update only part of a resource, not the whole thing. It’s similar to a PUT request, but the main difference is that PUT expects the entire payload, whereas PATCH lets us send only the fields we want to change. How to Test PATCH API Requests With Playwright TypeScript We’ll use the PATCH /partialUpdateOrder API from the RESTful e-commerce application that updates the available order using its order ID. Similar to the PUT API request discussed in the earlier section of this article, the PATCH API also needs an authentication token to update the order. As it is a PATCH request, we don’t need to send the full payload. Only the fields that require updates should be included in the request body. Test Scenario: Markdown ## Test Scenario Title: Partially update an existing orders in the system. ### Pre-condition: Valid orders are available in the system #### Test 1. Update the user_id and product_id for the order ID - 1 2. Verify that the Status Code 200 is returned in the response. 3. Assert that the order details have been updated correctly. The PATCH API request requires an authorization token. We’ll use the token from the auth.fixture.ts file that we generated in the earlier section while discussing the PUT API request. Let’s create a new spec file to partially update an order using the PATCH /partialUpdateOrder API. TypeScript # 6_patch-test.spec.ts import { test, expect } from "../fixtures/auth.fixture.js"; test("Update Partial Order using PATCH API Request", async ({ request, token, }) => { const updatedOrder = { user_id: "98", product_id: "214", }; const id = 1; const response = await request.patch( `http://localhost:3004/partialUpdateOrder/${id}`, { data: JSON.stringify(updatedOrder), headers: { "Content-Type": "application/json", Authorization: `${token}`, }, }, ); const responseBody = await response.json(); expect(response.status()).toBe(200); expect(responseBody).toEqual( expect.objectContaining({ message: "Order updated successfully!", order: expect.objectContaining({ user_id: "98", product_id: "214", }), }), ); }); Code Walkthrough Using the auth fixture for getting the token: The test uses a custom fixture that automatically provides a token, so we don’t have to call the POST /auth API every time. The token is generated before test execution and injected into the test context, ensuring secure and reusable authentication across multiple tests.Request body: The updatedOrder object contains only the fields user_id and product_id that need to be modified. As it is a PATCH request, there is no need to send the complete order payload; this makes the request lightweight and focused on partial updates.Sending the PATCH request: The request.patch() function is used to send a partial update request to the API endpoint, with the order ID dynamically added to the URL. The request includes the JSON-stringified body along with the Content-Type and Authorization headers. The token is passed in the Authorization header.Verifying the response: The response is parsed into JSON and validated to ensure a 200 status code is returned after the PATCH API is hit successfully. This verification ensures that the operation was successful. Then, expect.objectContaining is used to validate the response body, confirming that the success message and updated fields match the expected values. Test Execution To run the test, the following command should be executed from the terminal: Plain Text npx playwright test tests/6_patch-test.spec.ts The following screenshot of the test execution confirms that the test was run successfully: What Are DELETE Requests? A DELETE API request is used for removing a specific resource from the server. In most cases, it doesn’t return a response body. The resource is identified using a URI, and once the request is processed, it’s permanently deleted. Because of this, DELETE operations can have side effects, like removing records from a database. A DELETE request should be handled carefully because: Data deleted through a DELETE request is permanently removed and cannot be restored.It directly impacts the server data; hence, it is not considered a safe method.It may lead to inconsistencies if not managed appropriately.DELETE isn’t always idempotent in real-world scenarios. For example, the first DELETE request might return a 204 (No Content) indicating success, while calling it again on the same resource could return 404 (Not Found) because the resource has already been removed. How to Test DELETE API Requests With Playwright TypeScript We’ll use the DELETE /deleteOrder API endpoint from the RESTful e-commerce application for this demonstration. This API requires an order_id as a path parameter to identify the order to delete. A valid authentication token must be included in the headers. On successful execution, the order is removed, and a 204 (No Content) status is returned. Want to learn more about API Testing? Check out this tutorial to get hands-on API Testing Test Scenario: Markdown ## Test Scenario Title: Delete existing order from the system. ### Precondition - There should be some valid orders available in the system. #### Test Steps: 1. Delete the order with order_id "6" using the DELETE request. 2. Verify that 204 Status Code is returned in the response. To implement this test scenario, we need valid orders in the system, which can be created using the POST /addOrder API. Refer to the earlier tutorial on testing POST requests with Playwright TypeScript to explore different ways to generate request payloads and test the POST API request. Test Implementation Let’s create a new spec file to delete an order using the DELETE /deleteOrder API. TypeScript import { test, expect } from "../fixtures/auth.fixture.js"; test("Delete an order using DELETE API Request", async ({ request, token }) => { const id = 6; const response = await request.delete( `http://localhost:3004/deleteOrder/${id}`, { headers: { Authorization: `${token}`, }, }, ); expect(response.status()).toBe(204); }); Code Walkthrough Using the auth fixture for getting the token: The test imports a custom test from the auth fixture, which automatically provides a valid authentication token. This token is generated before the test runs and injected into the test context, so there’s no need to call the POST /auth API explicitly before the DELETE API.Sending the DELETE request: The request.delete() function is used to send a DELETE request to the API endpoint, with the ID dynamically passed in the URL to identify the order to be deleted. The request includes the Authorization header, where the token is sent to authenticate the request. No request body is required for this operation.Assertion: The test verifies that the response status is 204, which indicates that the request was successfully processed and the resource was deleted. Since DELETE APIs often don’t return a response body, validating the status code is sufficient to confirm success. Test Execution To run the test, run the following command from the terminal: Plain Text npx playwright test tests/7_delete-test.spec.ts The following screenshot confirms that the test ran successfully: A simple, practical approach is to call the GET API after running a DELETE request to confirm the resource has been removed from the system. Summary Testing PUT, PATCH, and DELETE APIs with Playwright and TypeScript helps ensure that the application handles updates and deletions correctly. While challenges such as managing test data, handling authentication, and maintaining test independence can arise, Playwright’s fixtures and built-in request handling make it easier to create clean and reusable tests. The main advantage is the ability to validate end-to-end API behavior with clear assertions and strong TypeScript support. In my experience, the tests should be isolated, avoid unnecessary dependencies between them, and validate both status codes and response data to make the API tests more robust and maintainable. Happy testing!!

By Faisal Khatri CORE

AWS Secrets Manager has a lovely feature: automatic rotation. You attach a rotation Lambda, set a schedule, and every 30 days your RDS password is rotated for you. Secure by default, no human involved. There's just one problem, and it's the one nobody mentions in the demo: your application has no idea it happened. Your Go service read SecretString once, at startup, and stashed the password in a struct. Thirty days later Secrets Manager quietly rotates it, the old value stops working, your connection pool starts throwing auth errors, and you find out via a pager alert at 3 am. The "fix" everyone reaches for is to redeploy on a schedule, or restart pods so the app re-reads the secret — which turns a security feature into an operational chore. mamori is a Go library I built to close exactly this gap. It watches AWS Secrets Manager (and SSM Parameter Store) and reconciles a rotation into your running process — typed, validated, redacted — so a rotation triggers a connection-pool rebuild instead of a restart. This is the AWS-focused companion to the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go. Read that first for the full model; here we go deep on AWS. The Two AWS Schemes The providers/aws module registers two ref schemes: RefReads fromaws-sm://<name>[#json-key]Secrets Manageraws-ps://<name>[#json-key]SSM Parameter Store A few real refs — note the triple slash on Parameter Store, because SSM parameter names are absolute paths that start with /: Plain Text aws-sm://prod/api-key # whole secret value aws-sm://prod/db#password # one key out of a JSON secret aws-ps:///myapp/log-level # SSM parameter (triple slash!) aws-ps:///myapp/db#password # one key out of a JSON SSM parameter The rule of thumb: secrets (passwords, tokens, keys) belong in Secrets Manager (aws-sm), and plain, non-sensitive config (log levels, worker counts, feature toggles) belongs in the cheaper Parameter Store (aws-ps). mamori lets you mix both in the same struct. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/aws Register it with a blank import for zero-config (it uses the standard AWS credential chain — environment, shared config, IRSA on EKS, EC2 instance role): Go import _ "github.com/xavidop/mamori/providers/aws" Or, when you need to pin a region or pass a custom client, construct the provider explicitly and hand it to Load/Watch with WithProvider — this takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) Your runtime's IAM role needs the obvious permissions, scoped tightly: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/*" }, { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/myapp/*" } ] } Load, Then Watch the Rotation Here's a service that pulls its database password from Secrets Manager and its operational knobs from Parameter Store: Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/aws" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` APIKey secret.String `source:"aws-sm://prod/api-key"` LogLevel string `source:"aws-ps:///myapp/log-level" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"aws-ps:///myapp/workers" default:"4" validate:"gte=1,lte=256"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // A rotation just landed — rebuild the pool, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) log.Println("database password rotated and applied live") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() log.Printf("started with %d workers at %s", cfg.Workers, cfg.LogLevel) } DBPassword and APIKey are secret.String, so they redact themselves everywhere — log.Printf("%+v", cfg) prints [REDACTED], json.Marshal emits "[REDACTED]", and slog logs them as [REDACTED]. The plaintext only ever escapes through an explicit .Reveal(). mamori even ships a go vet analyzer that fails the build if you put an aws-sm:// value into a plain string instead of a secret.String. Polling Done Right (Jitter, Batching, Atomicity) Secrets Manager has no push notification for value changes, so mamori polls — but it polls the way you'd want a fleet to poll: Jittered intervals. The poll interval is spread by ±20% by default, so 200 pods don't all hit GetSecretValue on the same tick and blow through your API rate limit. You can tune the spread with WithJitter.Change detection is cheap. mamori compares the secret's version (Secrets Manager's VersionId) before doing anything, so an unchanged secret costs one lightweight check, not a full re-decode.Batching. The AWS provider resolves multiple refs from the same account in as few API calls as possible, automatically — so a struct with ten aws-sm:// fields doesn't mean ten times the API traffic.Atomic + validated. When a rotation is detected, mamori re-validates the entire struct and only swaps it in if it's valid. OnChange never fires with a half-updated or invalid config. Compare this to the AWS Secrets Manager caching client: it caches and refreshes one secret at a time, with no struct composition, no validation, no diff-aware callback, and no config sources beyond Secrets Manager. mamori gives you the whole config surface — secrets and Parameter Store and env and files — in one typed struct with one reconciliation loop. Tolerate a Blip, Page on a Real Outage Secrets Manager occasionally throttles or has a transient error. You don't want that to page anyone — a blip should just retry with backoff and keep serving the last good value. But if Secrets Manager is genuinely unreachable for ten minutes, that's worth waking someone up. WithStale draws that line: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { pager.Page("AWS config stale >10m", stale) return } metrics.Inc("aws_transient_error") // blips: just a metric }), ) Mixing Sources in One Struct The real payoff shows up when your config isn't all in one place. mamori doesn't care that these live in four different systems — it's one struct: Go type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` // Secrets Manager Workers int `source:"aws-ps:///myapp/workers"` // Parameter Store LogLevel string `source:"env:LOG_LEVEL" default:"info"` // environment TLSCert []byte `source:"file:///etc/tls/tls.crt"` // local file } Wrapping Up Automatic rotation is only half a feature if your app has to restart to notice. mamori supplies the other half: it watches AWS Secrets Manager and Parameter Store and reconciles a rotation into your running Go service — typed, validated, redacted, no bounce. Shell go get github.com/xavidop/mamori/providers/aws Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early days, so feedback and issues are genuinely welcome. Happy coding!

By Xavier Portilla Edo CORE

Every Go service I have ever shipped ends up growing the same organ. It starts small: a couple of os.Getenv calls. Then a secret needs to come from AWS Secrets Manager, so you add the SDK. Then someone wants that secret to rotate without a deploy, so you add a time.Ticker. Then the ticker needs a mutex because two goroutines read the config while it is being replaced. Then a bad value slips in and takes down production, so you add validation. Then a secret shows up in a log line during an incident, and now you are adding a redaction wrapper too. By the end you have a bespoke ConfigManager in every repo: a loader, a fetcher, a refresh loop, a mutex, and a prayer. It is never quite the same twice, it is rarely tested well, and it is exactly the kind of code nobody wants to own. I got tired of writing it, so I built a library that does it once, properly. It is called mamori; it is open source, and this article is the full tour: what it is, why it exists, how to use it end to end, and when you should (and shouldn't) reach for it. The Restart-to-Rotate Problem Here is the assumption baked into most Go config code: configuration is static. You read it at boot, decode it into a struct, and treat that struct as immutable for the lifetime of the process. If something changes, you redeploy. That assumption was fine when "config" meant a log level and a port. It is not fine anymore. In a modern service, a meaningful amount of your config is dynamic by nature: Database passwords and API tokens rotate on a schedule (and increasingly, automatically).Vault hands you dynamic credentials with a lease that expires in an hour.Feature flags flip in the middle of the day, on purpose.TLS certificates get reissued. The industry answer to "config is dynamic" has mostly been to push the problem into the platform. On Kubernetes, the External Secrets Operator syncs your cloud secret into a Secret, and then... you still restart the pod to pick it up, or you mount it as a file and hope your app re-reads it. The dynamic part gets flattened back into a static one at the worst possible layer. mamori takes the opposite position: your process should be able to observe a change at the source and react to it, in memory, without bouncing. Rotating a password should rotate a connection pool, not recycle a pod. What Is mamori? mamori (守り, Japanese for "protection" or "safeguard") is an embedded Go library — not an operator, not a server, not a sidecar. It lives inside your process. Its tagline says it plainly: Typed, watchable config & secrets for Go. Load configuration and secrets from anywhere into validated Go structs — and keep them reconciled at runtime, without a restart. There are three concepts you actually touch, and they map cleanly onto three types: Refs. Struct fields carry a source tag containing a URL-like reference to where the value lives — env:LOG_LEVEL, aws-sm://prod/db#password, vault://secret/app#token. A ref names what you want, not how to fetch it.Providers. A provider resolves one scheme (aws-sm, vault, k8s-secret, …) into a value. Providers register themselves using the same pattern as database/sql drivers, so the core module carries zero cloud-SDK dependencies — you only pull in the providers you use.The Reconciler. This is the interesting part. Watch resolves every ref once (fail-fast at startup), then keeps watching each source. When a value changes, it re-validates the entire struct, and only if the new snapshot is valid does it atomically swap it in and hand you a diff-aware callback. Your app never sees a half-updated, invalid config. If you have used IOptionsMonitor<T> in .NET, w.Get() is the direct analogue: always the latest valid configuration, lock-free. How mamori Compares I want to be honest about prior art, because none of these primitives are new — mamori's contribution is composing them. gocloud.dev/runtimevar is the closest primitive. It watches a single variable beautifully, but it is one variable at a time, with no struct composition, no tags, no validation, no secret hygiene. mamori adds all of that on top of the same idea.Viper/koanf are excellent, but they are config-first with secrets bolted on. mamori is deliberately secrets-first, with config included: secret types redact by default, and there is a go vet analyzer that yells at you if you store a secret in a plain string.envconfig/caarlos0/env are load-once. No watching, env only.The External Secrets Operator is complementary, not competitive. mamori is for apps that want to skip the Kubernetes Secret hop entirely, or that do not run on Kubernetes at all. It keeps no persistent external state, so there is no finalizer lifecycle to babysit. And to be equally clear about what mamori is not: it is not a secrets store (no encryption at rest, no server), not a sync engine between stores, not a general feature-flag platform, and not cross-language. It is a Go-idiomatic library, and it stays in its lane. Getting Started The core module has built-in env:, file://, and dotenv: providers, so you can do something useful with zero extra dependencies: Shell go get github.com/xavidop/mamori Heads up: mamori requires Go 1.26 or newer. It leans on modern generics ergonomics, so this is a hard floor, not a suggestion. The whole API is driven by a struct and its tags. You describe what you want; mamori figures out how: Go package main import ( "context" "log" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` LogLevel string `source:"env:LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"env:WORKERS" default:"4" validate:"gte=1,lte=256"` TLSCert []byte `source:"file:///etc/tls/tls.crt"` } func main() { cfg, err := mamori.Load[Config](context.Background()) if err != nil { log.Fatal(err) } // cfg is fully resolved, defaulted, and validated. } The signature is exactly what you'd hope for: Go func Load[T any](ctx context.Context, opts ...Option) (T, error) Load resolves every ref once, applies defaults, validates, and returns your typed config. It fails fast: on any resolve or validation error, it returns the zero value and a non-nil error. It never returns a partially-populated struct — you either get a complete, valid config or an error. A few things worth knowing about the tags: TagMeaningsource:"..."The ref: where the value comes from.default:"..."Used only when the ref resolves to not-found — never on an error.validate:"..."go-playground/validator/v10 syntax, run on load and on every update.flatten:"json|yaml|env"Decode one payload into a nested struct.optional:"true"Tolerate not-found with no default (field keeps its zero value). That default-only-on-not-found rule is deliberate and important: if AWS Secrets Manager is having a bad day, you want an error, not a silent fallback to a default password. Watching and Reconciliation This is why mamori exists. Swap Load for Watch and your config becomes a living thing: Go func Watch[T any](ctx context.Context, opts ...Option) (*Watcher[T], error) You read the current config with a lock-free w.Get(), and you react to changes with an OnChange callback: Go w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(30*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { pool.Rotate(ev.New.DBPassword.Reveal()) } for _, f := range ev.Fields { log.Printf("%s changed: %s -> %s", f.Path, f.OldVersion, f.NewVersion) } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // always the last VALID config, no lock needed The Change[T] you get in the callback carries the old value, the new value, and a per-field diff, plus a Changed(path) helper so you only do expensive work (like rebuilding a connection pool) when the field you care about actually moved. The guarantees are the whole point, so let me spell them out: Atomicity. OnChange only ever fires with a fully re-validated snapshot. If an incoming value fails validation, the update is rejected: Get() keeps returning the last good config and OnError receives a *ValidationError. Your config never transitions into a broken state mid-flight.Push where possible, poll where not. Providers whose backend can push changes (Kubernetes informers, Vault lease watchers, Consul blocking queries, fsnotify for files) push natively. Everything else is polled on WithPollInterval, jittered by ±20% by default so a fleet of pods doesn't stampede your secrets backend on the same tick. If a native watch fails to start, mamori quietly falls back to polling.Coalescing. Rapid-fire changes within a debounce window (500ms default, overridable per-field with ?debounce=) collapse into a single event.Lease-aware. If a provider reports an expiry (Vault dynamic creds), mamori schedules a refresh before the lease dies, rather than waiting for the next poll.First event. Watch resolves the initial config before it returns, so OnChange fires only on subsequent changes. Read the initial state from w.Get(). There's a nice operational escape hatch too. Transient backend outages shouldn't page anyone — but prolonged staleness should. WithStale draws that line for you: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { alert.Page("config stale", stale) return } metrics.Inc("config_transient_error") }), ) A blip retries with exponential backoff and keeps the last good value. Ten minutes of failure escalates to a hard *StaleError you can actually page on. Secrets That Behave Like Secrets Configuration and secrets are treated differently, on purpose. Any field that holds a secret should be a secret.String or secret.Bytes instead of a plain string or []byte. These types redact themselves everywhere a value normally leaks: Go s := secret.NewString("hunter2") fmt.Println(s) // [REDACTED] fmt.Sprintf("%v", s) // [REDACTED] fmt.Sprintf("%#v", s) // [REDACTED] json.Marshal(s) // "[REDACTED]" slog.Info("login", "pw", s) // pw=[REDACTED] s.Reveal() // "hunter2" <- the only way to read the plaintext It works because the type implements fmt.Stringer, fmt.GoStringer, json.Marshaler, and slog.LogValuer. The single, greppable access point is Reveal(). That is a feature: in code review or an audit, grep Reveal shows you every place a secret is actually unwrapped. (secret.Bytes.Reveal() returns []byte rather than a string, for keys and certificates.) And because "just remember to use secret.String" is exactly the kind of discipline that erodes under deadline pressure, mamori ships a go vet analyzer that enforces it: Shell go install github.com/xavidop/mamori/tools/reconcilevet/cmd/reconcilevet@latest go vet -vettool=$(which reconcilevet) ./... reconcilevet flags any field whose source points at a secret-bearing scheme (aws-sm, gcp-sm, azure-kv, vault, op, sops, k8s-secret) but is stored in a plain string or []byte. The leak becomes a build failure, not a postmortem. Providers: More Than 30 Sources The core stays dependency-free; each cloud backend is its own module you opt into. A ref's scheme picks the provider: Shell go get github.com/xavidop/mamori/providers/aws # aws-sm:// aws-ps:// go get github.com/xavidop/mamori/providers/vault # vault:// go get github.com/xavidop/mamori/providers/gcp # gcp-sm:// go get github.com/xavidop/mamori/providers/azure # azure-kv:// go get github.com/xavidop/mamori/providers/k8s # k8s-secret:// k8s-cm:// go get github.com/xavidop/mamori/providers/consul # consul:// go get github.com/xavidop/mamori/providers/doppler # doppler:// go get github.com/xavidop/mamori/providers/onepassword # op:// go get github.com/xavidop/mamori/providers/sops # sops:// That is a small slice. All in, there are more than 30 providers: secret managers (AWS Secrets Manager, AWS Parameter Store, Vault, GCP, Azure Key Vault, Doppler, 1Password, SOPS), databases (Postgres, MySQL, SQLite, MongoDB, DynamoDB, Cosmos DB, Redis), KV stores (Consul, etcd), Kubernetes Secrets and ConfigMaps, Firebase/Firestore/Remote Config, object storage (S3, GCS, Azure Blob), and a whole shelf of feature-flag platforms (LaunchDarkly, Unleash, Flagsmith, ConfigCat, Split, GrowthBook, Flipt, GO Feature Flag). Registration follows the database/sql convention — a blank import runs the provider's init(), which calls mamori.Register(...): Go import _ "github.com/xavidop/mamori/providers/vault" When you need explicit configuration (a region, a custom client), construct the provider and pass it with WithProvider. That takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) The ref grammar is compact — scheme://path[#key][?opts] — with a couple of quirks worth internalizing: The #key fragment selects one field out of a JSON payload: aws-sm://prod/db#password.AWS Parameter Store, etcd, sops, and file preserve absolute paths, so they use a triple slash: aws-ps:///myapp/log-level, file:///etc/tls/tls.crt.Doppler is the one provider where the # fragment is required: doppler://backend/prd#STRIPE_API_KEY. When a payload is a structured blob, you can decode it straight into a nested struct with flatten: Go type Config struct { Redis RedisConfig `source:"aws-sm://prod/redis" flatten:"json"` } type RedisConfig struct { Addr string `mapstructure:"addr"` Password secret.String `mapstructure:"password"` DB int `mapstructure:"db"` } One AWS Secrets Manager call, one JSON blob, cleanly typed — with the password still redacting itself. Writing Your Own Provider If your source of truth is homegrown — an internal config service, a weird legacy database table — you can teach mamori about it. The interface is intentionally tiny: Go type Provider interface { Scheme() string Resolve(ctx context.Context, ref Ref) (Value, error) } A minimal implementation looks like this: Go func (p *Provider) Resolve(ctx context.Context, ref mamori.Ref) (mamori.Value, error) { raw, err := p.fetch(ctx, ref.Path) if isNotFound(err) { return mamori.Value{}, mamori.ErrNotFound // MUST satisfy errors.Is } if err != nil { return mamori.Value{}, err } if ref.Key != "" { raw, err = mamori.SelectKey(raw, ref.Key) // implements #key against a JSON payload if err != nil { return mamori.Value{}, err } } return mamori.Value{ Bytes: raw, Version: mamori.VersionHash(raw), // change detection; use a native revision if you have one Sensitive: true, // true for secret-bearing sources }, nil } func init() { mamori.Register(New()) } // database/sql pattern; panics on a duplicate scheme The Version field is what powers cheap change detection: if your backend exposes a revision (a VersionId, a Vault version, an mtime hash), return it; otherwise mamori.VersionHash(bytes) gives you a stable FNV hash for free. Two optional interfaces let you do better when the backend supports it: Go type WatchableProvider interface { Provider Watch(ctx context.Context, ref Ref) (<-chan Update, error) } type BatchProvider interface { Provider ResolveBatch(ctx context.Context, refs []Ref) (map[string]Value, error) } Implement Watch only if your backend can genuinely push — otherwise let mamori's poller do the work, so behavior stays consistent across providers. Implement ResolveBatch if you can fetch many refs in one round trip (the AWS provider does this automatically). Best of all, you don't have to trust yourself to get the contract right. There's a conformance kit — github.com/xavidop/mamori/providertest — that exercises resolution, not-found typing, version monotonicity, concurrency, context cancellation, native watching, goroutine hygiene, and even asserts you never accidentally log the payload. Point it at your provider, and it tells you if you've broken the rules. Middleware: Cache, Failover, Audit, Rate-Limit Because every provider is just a Provider, you can wrap one in another. The middleware package ships the decorators you'd otherwise reinvent: MiddlewareWhat it doesCache(ttl, inner)Memoize successful resolves. Errors and not-found are not cached.Failover(primary, replicas...)Try the primary, fall back to replicas on transport errors. Not-found is authoritative.RateLimit(rps, inner)Cap resolves per second so you don't trip your secrets backend's throttle.Audit(logger, inner)Log scheme, ref, latency, and outcome — never the value.Prefix(prefix, inner)Namespace every ref path — handy for multi-tenant setups. They compose by nesting, so you can read the stack top-down: Go import "github.com/xavidop/mamori/middleware" mamori.WithProvider( middleware.Cache(5*time.Minute, middleware.Audit(logger, middleware.RateLimit(10, middleware.Failover(primarySM, replicaSM), ), ), ), ) Crucially, the shipped middleware preserves the WatchableProvider and BatchProvider capabilities of whatever they wrap — decorating a provider never silently downgrades it from push to poll. Observability Out of the Box The core takes no OpenTelemetry dependency (keeping it lean), but there's a bridge module when you want metrics and traces: Shell go get github.com/xavidop/mamori/x/otel Go import ( "go.opentelemetry.io/otel" mamoriotel "github.com/xavidop/mamori/x/otel" "github.com/xavidop/mamori" ) meter, _ := mamoriotel.NewMeter(otel.Meter("mamori")) tracer := mamoriotel.NewTracer(otel.Tracer("mamori")) w, err := mamori.Watch[Config](ctx, mamori.WithMeter(meter), mamori.WithTracer(tracer), ) You get a mamori.resolve.duration histogram, mamori.refresh.count and mamori.watch.errors counters (all tagged by scheme), and a span per resolve. And because WithMeter/WithTracer take tiny internal interfaces, you can implement them against Prometheus, statsd, or a test recorder without pulling OTel into your binary at all. A Complete, Runnable Example Here is the whole loop — load, watch, react — in one file you can run today (it's the examples/basic program from the repo): Go package main import ( "context" "fmt" "log/slog" "os" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" ) const tokenPath = "/tmp/mamori-example-token" type Config struct { LogLevel string `source:"env:LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"env:WORKERS" default:"4" validate:"gte=1,lte=256"` APIToken secret.String `source:"file:///tmp/mamori-example-token"` } func main() { logger := slog.New(slog.NewTextHandler(os.Stdout, nil)) _ = os.WriteFile(tokenPath, []byte("initial-token"), 0o600) defer func() { _ = os.Remove(tokenPath) }() ctx, cancel := context.WithCancel(context.Background()) defer cancel() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(2*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { for _, f := range ev.Fields { logger.Info("config changed", "field", f.Path, "from", f.OldVersion, "to", f.NewVersion) } if ev.Changed("APIToken") { logger.Info("rotating clients with the new API token", "token", ev.New.APIToken) } }), mamori.OnError(func(err error) { logger.Warn("reconcile error", "err", err) }), ) if err != nil { logger.Error("watch failed", "err", err) os.Exit(1) } defer func() { _ = w.Close() }() cfg := w.Get() logger.Info("loaded config", "logLevel", cfg.LogLevel, "workers", cfg.Workers, "apiToken", cfg.APIToken) // Simulate a rotation a few seconds in. go func() { time.Sleep(3 * time.Second) _ = os.WriteFile(tokenPath, []byte("rotated-token"), 0o600) }() time.Sleep(8 * time.Second) logger.Info("final config", "apiToken", w.Get().APIToken) } Shell LOG_LEVEL=debug WORKERS=8 go run ./examples/basic Notice the token field in the log output: it prints [REDACTED] every time, even though the program is actively rotating it. That's the whole philosophy in one demo — live rotation, no restart, and nothing leaks. When to Reach for mamori (and When Not) Reach for it when: Your config or secrets genuinely change while the process runs, and you want to react in memory instead of bouncing pods.You pull from more than one source and are tired of a different SDK and refresh strategy for each.Secret hygiene matters and you want redaction and a go vet guardrail for free.You run outside Kubernetes, or you're on Kubernetes but want to skip the Secret-sync hop. Probably skip it when: Your config is a log level and a port from the environment, read once at boot. os.Getenv is fine — don't add a dependency.You need a secrets store (encryption at rest, a server component) — that's Vault's job, not mamori's.You're syncing secrets between stores — that's what ESO, vals, and teller are for. Wrapping Up mamori is the library I wish I'd had every time I typed type ConfigManager struct for the fifth time. It takes the boring, error-prone, security-sensitive plumbing — load, validate, watch, redact, reconcile — and makes it a typed struct and one function call. It's open source under the MIT license and currently at v0.1.0 — early, pre-1.0, and moving fast, so feedback and issues are genuinely welcome. Give it a spin: Shell go get github.com/xavidop/mamori Docs: mamorigo.dev/docsHomepage: mamorigo.devSource: github.com/xavidop/mamori Provider Deep Dives If you live on a particular platform, I've written focused, hands-on companions: Hot-reloading Kubernetes Secrets and ConfigMaps in Go — no pod restartsRotating AWS Secrets Manager secrets without restarting your Go serviceGCP Secret Manager, live-reloaded in GoAzure Key Vault secrets that reconcile themselves in Go If you build something with it, or write a provider for a backend I haven't covered yet, I'd love to hear about it. Happy coding!

By Xavier Portilla Edo CORE

Ever felt stuck writing Playwright tests because of messy or hardcoded test data? It usually starts simple, but as your tests grow, like testing an e-commerce or finance-related application, managing realistic data quickly becomes confusing and repetitive. More time is spent fixing test data than actually testing. That’s where the Builder Pattern, combined with Faker, can really help. It lets us create clean, flexible, and realistic test data with ease. In this article, we’ll learn how this approach can make the API tests with Playwright TypeScript easier to write, read, and maintain. What Is the Builder Design Pattern? The Builder Design Pattern is a creational design pattern that allows constructing complex objects step by step, allowing different representations of the same object. It separates object-creation logic from structure, making the code more readable and maintainable. This pattern is especially useful for objects with many optional parameters, as it avoids messy constructors and improves test data creation. What Is Faker? Faker is a library that generates realistic-looking fake data such as names, emails, addresses, numbers, and more. It is commonly used in testing to avoid hardcoding values and to simulate real-world data scenarios. By combining the Builder Pattern with Faker, we can generate realistic test data on the fly and perform testing without worrying about test data management, duplication, or maintenance. Faker can be installed using npm by running the following command in the terminal: Plain Text npm install @faker-js/faker --save-dev After installation, we can use the Faker library directly in the code by using the following import: TypeScript import {faker} from '@faker-js/faker'; How to Use the Builder Pattern With Faker in Playwright With TypeScript To use the Builder design pattern with Faker in Playwright TypeScript, the setup usually includes two main parts: ModelBuilder class The Model defines how the data should look, while the Builder class handles creating and customizing that data using Faker. Application Under Test We’ll use the POST /addOrder API of the RESTful e-commerce demo application for the demonstration. The following is the request payload of the API: JSON [ { "user_id": "string", "product_id": "string", "product_name": "string", "product_amount": 0, "qty": 0, "tax_amt": 0, "total_amt": 0 } ] Creating a Model for the Order The following model interface should be created for the Order payload: TypeScript export interface Order { user_id: string, product_id: string, product_name: string, product_amount: number, qty: number, tax_amt: number, total_amt: number } Code Walkthrough The Order interface represents the structure of an order object, where each field in the interface specifies the expected property and its data type: user_id, product_id, and product_name are strings.product_amount, qty, tax_amt, and total_amt are numbers. This interface ensures that any order object created in the code follows this exact structure, helping catch errors early and making the code more predictable and easier to maintain. Creating the Builder Class The following builder class is created to generate a new order with realistic fake values using the Faker library. TypeScript import { Order } from "../models/order.model.js"; import {faker} from '@faker-js/faker'; export class OrderBuilder { private order:Order; constructor() { const product_amount = faker.number.int({ min: 1, max: 100 }); const qty = faker.number.int({ min: 1, max: 5 }); const tax_amt = faker.number.int({ min: 2, max: 10 }); this.order = { user_id: this.generateUserId(), product_id: this.generateProductId(), product_name: faker.commerce.productName(), product_amount, qty, tax_amt, total_amt: (product_amount*qty)+tax_amt } } private generateUserId():string { return faker.number.int({min: 1, max: 50}).toString(); } private generateProductId():string { return faker.number.int({ min: 1, max: 100 }).toString(); } withUserID(user_id:string) { this.order.user_id = String(user_id); return this; } withProductId(product_id:string) { this.order.product_id=String(product_id); return this; } withProductName(product_name:string) { this.order.product_name = String(product_name); return this; } withProductAmount(product_amount:number) { this.order.product_amount= Number(product_amount); return this; } withQty(qty:number) { this.order.qty=Number(qty); return this; } withTaxAmt(tax_amt:number){ this.order.tax_amt=Number(tax_amt); return this; } withTotalAmt(total_amt:number) { this.order.total_amt= Number(total_amt); return this; } build():Order{ return this.order; } } Code Walkthrough The OrderBuilder class implements the Builder Pattern to create an Order object using Faker. The following is a detailed explanation of the code: Constructor(): The OrderBuilder class internally holds an Order object, which is based on the Order interface. In the constructor, it generates default random data using the Faker library.Private functions: The private functions, namely, generateUserId() and generateProductId(), are helper methods that generate random user IDs and product IDs, respectively.with() functions: The withX() functions, such as withUserId(), withProductId(), etc., allow overriding specific fields. Each method updates a value and “returns this,” enabling method chaining.Build functions: The build() function method finally returns the fully constructed Order object. In short, this class generates realistic test data by default while allowing selective customization, resulting in cleaner and more reusable test code. Writing the API Automation Test Using Playwright The following Playwright test demonstrates how to send a POST request using an order request payload generated through the Builder Pattern with the Faker library. TypeScript import { test, expect } from "@playwright/test"; import { OrderBuilder } from "../builders/order.builder.js"; import { Order } from "../models/order.model.js"; test("Add Order details using Builder Design Pattern", async ({ request }) => { const orderDetails = new OrderBuilder().build(); const orders: Order [] = [orderDetails]; console.log(orders); const response = await request.post("http://localhost:3004/addOrder/", { data: orders, headers: { "Content-Type": "application/json", }, }); expect(response.status()).toBe(201); }); Code Walkthrough The test imports OrderBuilder to create test data, and the Order interface to ensure type safety.Inside the test, new OrderBuilder().build() generates a complete order object with realistic values using Faker.The generated order is wrapped in an array, orders: Order[] = [orderDetails], as the API expects a list of orders. A console log statement is added to display the generated order in the console.A POST request is sent to the endpoint using Playwright’s request fixture, with the order payload passed in the data field and the appropriate Content-Type header.The test validates that the response status is 201, confirming that the order was successfully created. Test Execution Before running the test, start the RESTful e-commerce application using Docker by running the following command from the terminal: Plain Text docker compose -f docker-compose-ecommerce.yml up -d The test can be executed by running the following command from the terminal: Plain Text npx playwright test tests/post-api-builder-test.spec.ts The following is the output of the test execution: The order details generated using the Builder design pattern are printed on the console with fake data and are used as the request payload in the POST /addOrder test. Writing automated tests by combining generated fake data with selectively updated static values. In the previous test scenario, all order details were generated using the Faker library. However, in real testing scenarios, certain fields may require static values. The Builder Pattern supports this by allowing specific fields to be overridden with static data, while the remaining fields continue to be generated dynamically using Faker. TypeScript test("Add Specific Order details using Builder Design Pattern", async ({ request }) => { const orderDetails = new OrderBuilder().withUserID("56").withProductName("Donuts").build(); const orders: Order [] = [orderDetails]; console.log(orderDetails); const response = await request.post("http://localhost:3004/addOrder/", { data: orders, headers: { "Content-Type": "application/json", }, }); expect(response.status()).toBe(201); }); In this test, the user_id and product_name fields are explicitly set using the withUserID() and withProductName() functions. While the Builder generates the rest of the order details using the Faker library, these two fields are overridden with the static value provided in the test. Let’s execute the test again and check the output: In the above screenshot, it can be seen that the order details are generated successfully using the Faker library; however, the value for user_id is set to "56" and product_name is set to "Donuts," as these were explicitly provided as static values for those fields. Learn how to use the Builder Design Pattern in Java with a Datafaker library to generate test data for automated testing. Summary The Builder Design Pattern is really helpful for creating clean, flexible, and maintainable test data in API automation. When combined with Faker, it enables the generation of realistic data while still allowing specific fields to be customized as needed. There are multiple ways to supply test data in API automation for POST API testing, such as using JSON files or directly generating data with the Faker library in the request payload without the Builder Pattern. However, the choice of approach depends on the application under test, the level of flexibility required, and how the test framework is designed. Happy testing!

By Faisal Khatri CORE

Context The incident occurred in the production backend infrastructure of a mobile application. External HTTP traffic was handled by Traefik, deployed on Docker Swarm manager nodes. Traefik ran in swarmMode, used providers.docker, was configured to use the mgp_default network, and had lbswarm disabled so that routing used the required overlay network rather than the Swarm VIP. One of the critical services was mgp_router-app. It was an entry-point service with two replicas deployed to preserve availability if one of them failed. The service used placement constraints, max replicas per node = 1, and a Traefik health check configured for /actuator/health on port 8080. This meant we relied on external health checking and strictly controlled replica placement across nodes. Swarm-level load balancing was intentionally disabled. We needed routing control at the Traefik layer, along with the ability to quickly remove unhealthy backends based on their health check status. For this production setup, that provided more flexibility than the built-in Swarm load-balancing logic. What Happened The issue was neither a widespread outage nor a failure of the entire service. One replica became unreachable to Traefik over the overlay network: a Traefik instance on one of the manager nodes could not establish a connection to that replica’s backend endpoint. The second replica continued to work normally, so the service as a whole remained available. For regular traffic, the incident did not affect service availability: Traefik continued routing requests to the healthy replica and removed the unavailable backend from routing. During targeted manual testing of the unhealthy replica’s backend endpoint through Traefik, requests ended in a timeout or HTTP 503. The container appeared healthy, but only locally. The service responded inside the container, and /actuator/health returned UP. However, other containers and Traefik instances on remote nodes could not establish a connection to that replica through its IP address in mgp_default. Initial Checks The container had the same DNS name in both networks. This ruled out name mismatch as an independent cause of the unavailability: the issue occurred specifically at the network-connectivity level to the replica IP in mgp_default. First, we identified the problematic container and inspected its state: Shell docker ps --filter "name=mgp_router-app" --q docker inspect 6bd15b62f891 Next, we checked the networks connected to the container: Shell docker inspect 6bd15b62f891 --format '{{json .NetworkSettings.Networks}' | jq The container was attached to two networks: ingress with IP 10.22.3.137/22;mgp_default with IP 10.29.72.232/13. We then entered the container and checked whether the application was listening on the expected port: Shell docker exec -it 6bd15b62f891 bash ss -tuln The health endpoint also responded correctly: Shell curl "http://10.29.72.232:8080/actuator/health" The expected response was: Shell {"status":"UP"} At this point, it was clear that the issue was not the application itself, its startup, or its bind address inside the container. The service was alive, but that alone said nothing about its reachability from outside the container. Checking Connectivity Between Nodes The next step was critical: We checked whether another node in the cluster could reach the container using its overlay IP. Shell docker exec -it 373354916bfc bash ping 10.22.3.137 This was the IP from the ingress network, and the test confirmed ICMP reachability of the address in that network. That indicated the issue did not look like a complete loss of inter-node connectivity. The situation was different for the mgp_default IP: pinging it from other nodes failed, even though this overlay network was supposed to provide inter-container communication in Docker Swarm. This distinction was important. ICMP reachability of the address in ingress remained available, while connectivity to the replica through mgp_default was broken. Therefore, the problem was not the container itself or a complete network failure; it was a specific break in the network path used to reach that replica. What Docker Logs Showed The dockerd logs on the affected node provided another useful signal. They contained repeated messages related to Swarm network synchronization: Shell Jun 02 11:08:42 node-router2 dockerd: time="2025-06-02T11:08:42.462965233Z" level=warning msg="bulk sync to node 091d01df89ae failed: failed to send a TCP message during bulk sync: dial tcp 10.170.0.145:7946: i/o timeout" Jun 02 11:17:02 node-router2 dockerd: time="2025-06-02T11:17:02.581936915Z" level=error msg="Bulk sync to node 091d01df89ae timed out" The logs also contained NetworkDB stats entries: Shell Jun 02 11:07:08 node-router2 dockerd: time="2025-06-02T11:07:08.012509772Z" level=info msg="NetworkDB stats node-router2(2642aeb5172b) - netID:q9athb3moj967oxmb3y8xw4lj leaving:false netPeers:58 entries:13563 Queue qLen:2 netMsg/s:7" These messages do not prove a specific bug or root cause, but they indicate problems with exchanging network state between nodes. Combined with the endpoint being unreachable through mgp_default, they strengthened the working hypothesis of an overlay-networking issue. This was an important diagnostic signal: the Docker daemon was running, and the container was started, but the cluster’s network state was not ideal and was likely becoming inconsistent during node-to-node state exchange. Investigating the Hypotheses Hypothesis 1: The container itself was the problem. This hypothesis was not confirmed. A curl request inside the container to http://10.29.72.232:8080/actuator/health returned {"status":"UP"}, and ss -tuln showed that the service was listening on the required port. Hypothesis 2: The problem was port publishing or binding. This hypothesis was not confirmed. Even though port 10005 was reachable externally, that did not explain why Traefik could not reach port 8080 over the overlay network. The container worked locally, while the issue occurred on the network path between nodes. Hypothesis 3: Traefik itself was the problem. Probably not. Traefik was not the source of the failure; it was the first component that detected that the backend was unavailable from its node. Hypothesis 4: The Docker Swarm overlay network was the problem. This became the primary working hypothesis. The replica was reachable from inside its container but was not properly reachable from other nodes through its overlay IP. Shell curl http://10.29.72.232:8080 nc -vz 10.29.72.232 8080 The curl request worked locally, while nc -vz 10.29.72.232 8080 failed from other nodes. This indicated that traffic was not being routed correctly through the overlay network. An MTU/VXLAN issue was also considered. This is a plausible class of problems for Docker overlay networks, especially when there is an MTU mismatch or a Path MTU Discovery issue. In this case, however, it remained only a hypothesis rather than a confirmed root cause. Recovery We restored the problematic replica by forcing Docker Swarm to recreate the task through a rolling update: Shell docker service update --force \ --update-parallelism 1 \ --update-order start-first \ --update-delay 30s \ --detach=true \ mgp_router-app --force triggered an update even though neither the image nor the service configuration had changed. Swarm recreated the task for the affected replica and reattached it to mgp_default; after that, Traefik could reach it again over the overlay network. The rollout parameters were selected to avoid affecting service availability: --update-parallelism 1 – update one replica at a time;--update-order start-first – start the new task before stopping the old one;--update-delay 30s – give the new task time to start and pass its checks. Restarting Docker on the node and placing the node into drain mode were considered fallback measures, but they were not required to resolve this incident. Why It Happened The exact upstream root cause cannot be stated without additional evidence. However, the most well-supported explanation is that a local network-state inconsistency occurred in the Docker Swarm overlay network. As a result, one replica remained operational inside its container, but became unreachable through its overlay IP. The most plausible explanations include: the overlay network on the node was not fully ready when the container started;an issue in Docker libnetwork;an outdated VXLAN routing table;an iptables issue or a race condition while updating network rules;a node restart without restarting the Docker daemon;an issue with a Swarm endpoint that was not properly advertised or connected. These situations are particularly unpleasant because Docker continues to consider the container running and its internal health check green, while the network path to the replica is already partially broken. Key Takeaways The main conclusion is simple: a healthy container does not necessarily mean a reachable replica. This is critical in production because: a local curl inside the container does not prove external reachability;an internal health check does not guarantee that Traefik can reach the backend;Swarm may remain formally healthy even when one replica has dropped out of normal network connectivity. Therefore, in such clusters, it is necessary to check not only container state but also inter-node reachability over the overlay network. What to Monitor To prevent similar incidents, it is useful to monitor not only service health but also the cluster’s network plane. The minimum set of checks includes: reachability of container overlay IPs from other nodes;health checks through Traefik;bulk sync and NetworkDB errors in Docker logs;recurring issues on the same node;consistency between a container’s local health and its external reachability. It is also useful to have a script that specifically checks connectivity between nodes through the overlay network, not just through ingress or localhost. Conclusion This case demonstrated several important points. First, a container can be healthy locally while remaining unreachable externally. This is especially important for services that handle user traffic. Second, Traefik is a good early indicator of network problems. It can reveal that a backend on one node has stopped responding. Third, Docker Swarm overlay networking can degrade locally. In that case, the cluster may appear operational, while one replica has already dropped out of normal routing. Fourth, such incidents must be diagnosed along the network path, not only through application state. First, check the container from the inside, then verify reachability from other nodes, then inspect Docker logs, and only after that draw conclusions about the root cause.