Inside terraform-provider-archive: A Memory Pattern From 2016 That Scales With Your Lambdas

A CI Runner That Shouldn't Have Died

If you deploy AWS Lambdas through Terraform, you almost certainly use archive_file. With enough lambdas, a single terraform apply can kill the CI runner with OOM. The trickiest part is that you will not see any errors in Terraform output and have no clue what just happened.

I noticed this when my lambdas started failing — every first terraform apply after a routine change. SIGKILL from the kernel OOM killer and nothing in Terraform logs. The strange part is that reapply sometimes worked — not always on the first try, but eventually it went through. I've named the ticket "Flaky CI," and two weeks of investigation was focused on the CI itself: runner memory, parallel jobs, Docker leaks. terraform apply was the last suspect — from my perspective, there was no way or reason for it to consume so much memory.

If you've never wondered how Terraform providers work, it's actually pretty simple. Most of them are just API wrappers. They send HTTP requests, parse responses, and update state. archive_file is one of the exceptions — it works with real files on disk. This means that its memory usage is actually determined not by the number of defined resources, but by the total size of the data it should process. That's why the pattern went unnoticed for years — without knowing about the provider's insides, the issue looks like some CI flakiness.

When I finally reached the source code, the answer was found in a few lines in zip_archiver.go file.

What archive_file Actually Does

archive_file data source creates a zip or tar archive from a directory or file. This is a standard pattern for lambdas: you point source_dir at the function code and pass the resulting archive to aws_lambda_function.

data "archive_file" "lambda" {
  type        = "zip"
  source_dir  = "${path.module}/src"
  output_path = "${path.module}/lambda.zip"
}

Nothing suspicious at first glance, but behind these lines is a call chain, which is worth a deeper look.

When Terraform processes this data source, the provider calls archiveFile — it creates a ZipArchiver and iterates over files in source_dir. For each file, it calls the ArchiveFile method, which does the following:

content, err := os.ReadFile(fname)
// ...
f, err := a.writer.Create(name)
// ...
_, err = f.Write(content)

os.ReadFile reads the entire file into a []byte — one contiguous buffer in memory. Then that buffer is passed to the zip writer via Write. After the write, the buffer becomes garbage.

This was a design choice from 2016, and at the time, it was reasonable. Terraform configurations archived small files — configs, scripts, and templates. A typical source_dir weighed something like kilobytes, so there was nothing to optimize at this point. That's why the simplest way to read a file was chosen — os.ReadFile. The code looks like a textbook example.

But the context changed. Lambda zips today are 50-250 MB uncompressed. ML models, large dependencies (numpy, pandas, puppeteer), bundled assets. And teams deploy not one lambda but five, ten, or twenty through a single Terraform workspace. The code from 2016 didn't change. The scale of the data did.

Why Can't the Garbage Collector Help

The natural and reasonable question: doesn't Go's garbage collector reclaim memory between files? GC runs indeed — it just has nothing to reclaim.

All ten archive_file data sources are independent — they have different source directories and no shared references (if you do not specify them directly). Terraform's graph walker places them at the same level and evaluates them concurrently. This is usually a good thing timewise, but not in this case, as all 10 buffers are alive at the same time.

Each goroutine holds its 50 MB until zip write completes. The garbage collector scans the heap and identifies every buffer as still in use, so it reclaims nothing. Meanwhile, peak heap hits 10 x 50 MB = 500 MB (measured: 508 MB).

If the model is right, peak memory should scale linearly with parallelism. Your CI runner's memory limit doesn't.

Measuring the Pattern

I've chosen two ways of measurement: a standard Go benchmark for precision (isolating the archiver) and a Terraform integration test for realism (a real provider during terraform plan).

The headline: for 10x50 MB concurrent archives, peak heap drops from 508 MB to 8 MB -- a 98% reduction.

Full results, heap growth during archiving, buffered versus streaming:

• 1x50MB:  50.8 → 0.8 MB  (98% reduction)
• 10x10MB: 108  → 8.1 MB  (92% reduction)
• 10x50MB: 508  → 8.1 MB  (98% reduction)

Real Terraform under terraform plan with parallelism matrix, peak RSS in MB:

Buffered RSS scales linearly with parallelism. Streaming flattens the curve.

One anomaly you could've noticed: at p=1, streaming shows a higher RSS than buffered. I'm fairly sure it's just noise. Single-archive runs finish fast, and sampling RSS every 100ms is too coarse to catch what's really happening in that window. The number that matters is p>=2, and that's where the pattern holds.

On speed: Go benchmark wall time stays within about 3% across every scenario. So the streaming fix isn't quietly buying memory savings with a performance hit. You get the memory back for free.

All measurements are reproducible: https://github.com/olegmmv/terraform-archive-memory-research.

Putting these measurements together gives a three-stage picture of the memory cost:

The third row isn't measured, but is arithmetic. I'll describe later why, but for now, just keep in mind that it shows what we'd see if a second os.ReadFile in the output path is also streamed.

The Tar Archiver Already Streams

The fix isn't speculative; just open a neighboring file in the same provider. In tar_archiver.go, addFile opens the file, defers close, and copies via io.Copy into tarWriter. No buffering — streaming by default.

file, err := os.Open(filePath)
// ...
defer file.Close()
// ...
_, err = io.Copy(a.tarWriter, file)

The zip_archiver.go path, though, chose the buffered approach:

content, err := os.ReadFile(infilename)
// ...
_, err = f.Write(content)

Same codebase and job to be done, but two different choices. archive/zip.Writer.Create returns an io.Writer that streams, with CRC-32 computed during the write via crc32.NewIEEE. There was never a technical barrier.

The only thing needed for the fix now is applying the same pattern.

The Streaming Fix

Here is the diff: replace os.ReadFile with os.Open and Write with io.Copy:

-  content, err := os.ReadFile(infilename)
+  file, err := os.Open(infilename)
   if err != nil {
     return err
   }
+  defer file.Close()

   if err := a.open(); err != nil {
...
-  _, err = f.Write(content)
+  _, err = io.Copy(f, file)

Everything else stays the same; the only thing that's different is the read-write pattern. This is the actual implementation behind the streaming numbers in the previous section.

The streaming version does still allocate memory, of course — you can't get to zero. But it's way down: my benchmark put it at around 0.8 MB. This is due to archive/zip internal buffering: the io.Copy buffer, the deflate compressor state, and small zip metadata structures.

One caveat worth flagging: this is the input side only. On the output path, the provider uses its own ReadFile function to compute checksums on the completed zip archive.

The Second ReadFile: Output Checksums

The Go benchmark showed a 98% reduction, but terraform plan with parallelism=10 only drops from 1034 MB to 533 MB -- about 50%. Where's the missing 48%?

Once the zip lands on disk, the provider turns around and reads it straight back. That's what genFileChecksums does: it opens the output file and computes four hashes -- md5, sha1, sha256, sha512 -- for Terraform state. And each one of those hashes wants the full file content. So the provider pulls the entire output zip into memory, using the same os.ReadFile we've been dealing with all along.

In my benchmark, the output zip comes out roughly the size of the input. The test data is random bytes, and Deflate can't do much with those. Real Lambda packages compress a lot better, but the pattern remains: the provider reads whatever the output size is back into memory. Run ten of these in parallel at 50 MB a pop, and you're already 500 MB deep, purely on checksums.

The PR goes after the input side. It removes the os.ReadFile allocation during archive creation, and the effect is big. In straight Go benchmarks, heap usage drops by 98%, from 508 MB to 8 MB. Real Terraform runs are tamer, about half: peak RSS falls from 1034 MB to 533 MB. So where's that remaining 533 MB coming from? It's the second os.ReadFile, the one inside genFileChecksums, still reading the finished zip back into memory so it can hash it for Terraform state.

Technically, you can stream the checksums too. hash.Hash already satisfies io.Writer, so nothing stops you from wrapping all four hashes in an io.MultiWriter and feeding them while the zip is being written. One pass, no second read.

The catch is that it's a very different patch from the input-side one. genFileChecksums is structured around post-hoc reading. Making it streaming means restructuring how the provider integrates checksum computation with archive creation. That's state-management territory, not plain I/O.

If both sides streamed, the only thing left to allocate would be io.Copy's default buffer. Ten goroutines, 32 KB each, and you land at 320 KB total. Throw in a sliver of zip writer state per goroutine, and that's basically it. The theoretical floor.

What the PR actually does is the first half: input streaming, leaving that 533 MB residual behind. The output half, streaming through MultiWriter, is written down as future work. So one PR cuts the problem in half. Closing it out takes two.

What It Costs in Practice

At the Lambda deployment limit of 250 MB, ten concurrent archives push peak heap to roughly 5 GB -- well past most CI runner allocations.

There are workarounds, each with a price tag. Dial parallelism down, and you trade throughput for memory. Spin up beefier CI runners, and you trade dollars for memory. Both get you unstuck, but neither addresses the root cause.

The PR is up at https://github.com/hashicorp/terraform-provider-archive/pull/501.

The fix is under ten lines of Go, so the investigation took much longer than the implementation.

Some design choices age well, but some scale with your infrastructure.