Replacing Direct Storage URLs With a Media Proxy at Scale

The first enterprise client to use our automated reporting feature filed an escalation. Every image in their weekly email was a broken red X, every single one. No thumbnails rendered, so the report was effectively unusable.

There was no bug. The code did exactly what I'd designed it to do months earlier, which was the problem. I had stored raw third-party storage URLs in our database and passed them straight to the frontend. Image tags, video tags, email templates — all of them pointed at someone else's domain. It worked in browsers. Then it hit a corporate email client with a strict image-domain allowlist. Our third-party hosts looked like tracking domains, so the client blocked every image.

That was the first failure mode. The second showed up when an external media provider purged a batch of old assets from its CDN and thousands of images in our app broke overnight. The third came from a security review that flagged our public bucket URLs. We were running a multi-tenant product where customer data isolation was a hard requirement, and a public bucket clearly failed that test.

The scale mattered. The serving layer I owned had grown past a few hundred million external media files, north of a hundred terabytes, with hundreds of thousands of new files landing daily. At that volume, there's no such thing as an edge case. A 0.01% breakage rate sounds like rounding error until you do the math and realize it's tens of thousands of broken images sitting in front of paying customers. So I rebuilt it.

Why Storing Raw Media URLs Fails

Here is why the store-the-URL, serve-the-URL pattern breaks down in production.

You Don't Control Access

Point your frontend at a third-party URL or a raw bucket path, and you've given up your access control layer. Your options are either a public bucket, which gives up access control entirely, or short-lived signed URLs (ref) that expire on a schedule that now becomes part of your application contract and breaks cached pages, bookmarks, and emailed links.

Email Clients Silently Destroy Your Images

This failure mode is easy to miss if you test in browsers and not in corporate mail clients.

Corporate email clients keep allowlists of trusted image domains. Embed report images hosted on <storage.thirdparty.com> and the client blocks them (ref) silently. No HTTP error, no bounce, nothing in your logs. The recipient sees blank squares, and you see a perfectly healthy dashboard.

I burned two days on this. Two days convinced me it was a rendering bug because the images loaded fine in every browser I tried. The mail client was the one saying no. Once I traced it to domain allowlists, the requirement changed: the client had to request media from our domain, not a storage host.

Dead Links are Someone Else's Decision

Storing a third-party URL as your canonical reference means trusting an external host to keep that exact path alive forever. They won't. Ad platforms rotate assets on their own schedule. A CDN vendor will deprecate an endpoint mid-migration and storage providers sunset whole APIs on their own timeline, taking your stored paths down with them.

Each of those events 404s your stored URL with zero warning and zero fallback. You can't monitor hundreds of millions of external URLs for availability. At production scale, the stable option is to ingest the bytes and stop treating third-party URLs as durable identifiers.

The Fix: Make It a Routing Problem

Here's the reframe that unlocked the design. We had been treating media serving as a storage problem. It was really a routing problem.

We put an API gateway at the edge, a custom proxy service behind it, and kept storage details out of every client. The client never sees a bucket name, a storage path, or a signed URL. The client knows one thing: a UUID-backed endpoint on our own domain.

https://api.ourdomain.com/media?action=display&id=<uuid>

The proxy service authenticates the request, resolves the UUID against a distributed relational database to get the internal blob ID, hands the blob ID and headers back to the file service, which fetches and streams the bytes. That file service reads customer blobs with delegated privilege tokens which are scoped and short-lived.

Now the email client sees our domain, not some storage host it doesn't trust. Access control lives in one place. And when storage changes later, we swap it behind the proxy instead of rewriting frontend URLs or breaking customer integrations.

The Metadata Layer

Routing needs a fast lookup table. Every asset lives as a row in a distributed relational database, picked for strong consistency and low-latency reads across regions. These lookups sit in the hot path of every page render and every report, so this choice matters.

CREATE TABLE MediaAsset (
    media_uuid         VARCHAR(36) NOT NULL,
    tenant_id          VARCHAR(50) NOT NULL,
    source_system      VARCHAR(50),
    media_blob_id      VARCHAR(255) NOT NULL,
    thumbnail_blob_id  VARCHAR(255),  -- preview JPEG for videos
    media_format       VARCHAR(20),   -- e.g., 'IMAGE' or 'VIDEO'
    created_at         TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (media_uuid)
);

The proxy queries by UUID, pulls media_blob_id and media_format, and hands the blob reference to the file service with the scoped token. The UUID is the only identifier the outside world touches. Bucket, region, and replication tier all stay private.

One deliberate call: The primary key is a high-entropy UUID, not a time-sortable one. In a single-node database with a clustered B-tree index, this would be a mistake; high-entropy keys fragment the index, and you'd reach for something time-sortable like UUIDv7 or ULID instead. In a range-sharded distributed database, the logic flips. Time-ordered keys pile every insert onto the tail shard and create a write hotspot. A high-entropy UUID spreads inserts across the keyspace, which is what lets parallel pipeline workers write without piling onto one shard. The section below shows how the UUIDs are computed. 

The Ingestion Pipeline: How the Bytes Get There

So far we’ve only talked about the serving path. The other half of the system is the ingestion pipeline that pulls media in, stores the bytes, and generates the thumbnail data the next section depends on.

When an upstream system surfaces a new asset, say an ad platform publishes an image or a partner API pushes a video reference, a task lands in a queue. Distributed workers pull from it. Per asset, a worker:

1. Fetches the raw bytes from the external URL while it's still live.
2. Detects the format (image vs. video, MIME type, resolution) from file headers.
3. Writes the bytes to internal object storage and gets back a 'blob_id'.
4. Inserts the metadata row: UUID, blob ID, detected format, and thumbnail_blob_id = null if thumbnail generation has not happened yet. 

Then an async enrichment pass, only when needed:

     5. Generates derived assets. For video, that means extracting a preview frame into a separate JPEG blob, the thumbnail_blob_id. Pro tip: don't grab the literal first frame; it's a black fade-in on most videos. Sample from around the one-second mark instead.

     6. Updates the metadata row with the thumbnail_blob_id.

The UUID is derived deterministically by hashing three inputs together: the tenant, the source system, and the external asset ID. That gives us two properties at once. First, the same upstream asset always maps to the same UUID no matter how many times its task message gets delivered, which is what makes idempotency work. Second, the hash output is high-entropy enough to spread inserts across the keyspace instead of piling them onto one shard. Folding the tenant into the hash means two customers ingesting the same upstream asset get different UUIDs and separate rows, which is what allows for tenant-scoped lookup in the routing logic.

In practice, the same external asset shows up in multiple task messages all the time. Retries after transient failures, duplicate events from upstream. The worker computes the UUID, checks whether the row already exists, and skips if it does. This keeps the blob store free of duplicate copies without distributed locking.

Parallel workers can insert metadata rows concurrently with no central sequence to fight over. The pipeline writes, the proxy reads, and the UUID is the contract between them.

I got one thing wrong in the first version: fetch-and-store and thumbnail generation ran in the same worker, so one slow video frame extraction would block the queue behind it and delay plain image ingestion. Splitting that into two stages: a fast store pass and an async enrichment pass cuts image ingest latency by more than half.

That split opens a window where an asset is servable, but its thumbnail_blob_id is still null because enrichment has not caught up yet. The proxy handles that state by returning a neutral placeholder image instead of a 404. Browsers render a placeholder, and the real thumbnail shows up on the next request after enrichment lands.

What the Proxy Enabled in the Serving Layer

Once the proxy was in place, several frontend workarounds turned into normal server-side code paths. The underlying problem was the serving path. We just hadn't isolated one before.

Video Thumbnails

Our app shows dense grids of assets, lots of them videos. Loading full video files to paint a preview grid is wasteful, and the client-side workarounds (lazy loading, placeholders, canvas frame extraction in the browser) all fell over at our data volume. On a 50-video grid, preview rendering was visibly slow.

With the proxy in the middle, one 'action' parameter was all we needed. The frontend asks for action=thumbnail, the proxy service skips the video entirely and returns that pre-generated preview JPEG the ingest pipeline already stored. It becomes the <video> poster attribute. Grid paints instantly, video loads on play.

Before this, frame extraction happened in the user path, over and over for the same video. Now it happens once at ingest and every viewer reuses the result. With repeated views at this scale, moving frame extraction off the request path was not a micro-optimization.

Bulk Downloads

Users often needed dozens of assets at once, sometimes hundreds for offline review or handoff. Before the proxy, the frontend had to fan out into many concurrent downloads. Browsers started warning, connections saturated, and individual files timed out without a useful error.

After the proxy was in place, bulk download became a normal code path instead of a frontend juggling act. The client sends multiple UUIDs on the download action, the proxy resolves them, and the file service streams the blobs into a single zip archive. It writes each blob into the archive as it arrives rather than buffering the whole thing in memory, so memory stays flat whether the user asked for five files or five hundred.

Two costs come with streaming, and you should know you're paying them. First, there is no Content-Length header up front (MDN on the Content-Length header), so the browser can't show a reliable download progress bar. Second, since the 200 status goes out when the stream opens, a blob fetch failing mid-stream can't change the response code. The connection just drops, and the client sees a truncated zip. That was an intentional trade-off: worse progress signaling in exchange for predictable memory usage. It makes sense when large downloads are common, and memory pressure matters more than perfect browser UX.

Simplified routing logic:

def handle_media_request(request):
    action = request.query_params.get('action')
    uuids = request.query_params.get('id').split(',')
    uuids = list(dict.fromkeys(uuids))  # dedupe, preserve request order

    # ONE batched lookup, scoped to the caller's tenant. The WHERE clause
    # filters on tenant_id, so a UUID owned by another tenant simply isn't
    # in the result set. This is the authorization check.
    rows = db.get_media_metadata_batch(uuids, tenant_id=request.user.tenant_id)

    # Any requested UUID we didn't get back is either unknown or cross-tenant.
    if len(rows) != len(uuids):
        return NotFound()

    by_uuid = {row.media_uuid: row for row in rows}

    # Token scoped to exactly the blobs we're about to serve, not the whole
    # user. Short-lived, travels with every file service call below.
    blob_scope = [r.media_blob_id for r in rows] + \
                 [r.thumbnail_blob_id for r in rows if r.thumbnail_blob_id]
    auth_token = generate_delegated_access_token(request.user, blob_scope)

    if action == 'display' and len(uuids) == 1:
        meta = by_uuid[uuids[0]]
        return FileResponse(
            blob_id=meta.media_blob_id,
            auth_token=auth_token,
            headers={'Content-Type': get_mime_type(meta.media_format)}
        )

    elif action == 'thumbnail' and len(uuids) == 1:
        meta = by_uuid[uuids[0]]
        if meta.thumbnail_blob_id is None:
            # Enrichment pass hasn't landed yet, serve a placeholder
            return FileResponse(
                blob_id=PLACEHOLDER_BLOB_ID,
                auth_token=auth_token,
                headers={'Content-Type': 'image/jpeg'}
            )
        return FileResponse(
            blob_id=meta.thumbnail_blob_id,
            auth_token=auth_token,
            headers={'Content-Type': 'image/jpeg'}
        )

    elif action == 'download':
        if len(uuids) == 1:
            meta = by_uuid[uuids[0]]
            return FileResponse(
                blob_id=meta.media_blob_id,
                auth_token=auth_token,
                headers={
                    'Content-Disposition': f'attachment; filename={uuids[0]}'
                }
            )
        else:
            # File service streams these blobs into one zip on the fly
            blob_ids = [by_uuid[u].media_blob_id for u in uuids]
            return ZipArchiveResponse(
                blob_ids=blob_ids,
                auth_token=auth_token,
                headers={'Content-Type': 'application/zip'}
            )

    return BadRequest()  # unknown action, or display/thumbnail with >1 id

Production Notes

The proxy has been running in production for over a year. Broken email images, dead links, and access control gap issues are fully resolved. A few things I'd flag for anyone building something similar:

1. Cache the UUID-to-blob mapping. The media_uuid --> media_blob_id mapping is immutable, and a 60-second in-memory TTL gave us a 95%+ hit rate. Why only 60 seconds for immutable data? Because immutable isn't permanent. Assets get deleted for takedowns and customer offboarding, and the TTL bounds how long a deleted asset can still be served from cache. I skipped caching initially and paid for it with tail latency spikes once traffic grew.
2. Separate your ingest stages. Fetching bytes and generating thumbnails in the same worker means one slow video frame extraction blocks the whole queue. Do the fast store step first, then generate thumbnails asynchronously. If enrichment is still catching up, serve a placeholder instead of failing the request.
3. Size your connection pools before you need to. The proxy holds connections to both the database and object storage, and under sustained load, the default pool sizes ran out. Requests queued, timeouts triggered retries, and the retries added more pressure to the same bottleneck. It is a classic incident loop that never shows up in design docs.
4. If you expect media volume to grow, put the proxy in before direct URLs spread through every client. The extra hop cost us a few milliseconds, but it gave us one place to enforce auth, serve thumbnails, package downloads, and hide storage migrations. Retrofitting that after a customer incident is far more expensive.