Prompt, Fine-Tune, or Compile: The Three Ways to Build Anything in AI

It started as a fleeting thought while I was heads-down building agentic AI systems: somewhere between "just call the API" and "let's train our own model," we've quietly ended up with three completely different ways to solve the same problem. Most teams treat that as a single decision, made once, early, and never revisited.

It isn't. It's a portfolio you manage for the life of the product.

Here's the framework, and why I think most teams have the sequencing backward.

The Three Tiers

1. Model API reliance. You call the frontier model, Claude, GPT, Gemini, whichever lab is ahead this quarter, and let its R&D absorb the part of the problem you don't understand yet. This is the right default when you genuinely don't know the shape of the task: when "correct" is still being defined, when volume is low, when the fastest way to learn is to ship and watch what breaks.

2. Fine-tuning open-source models. Once a use case turns out to be repeatable, same shape of input, same shape of output, high enough volume that you're paying real money for it every month, you stop renting intelligence and start owning it. You fine-tune an open-weight model on your own data. You don't have to chase every new open-source release to stay current; you can do this on a slow, deliberate cadence while gradually weaning that specific use case off the frontier API.

3. Migrating to declarative software. Eventually, for the use cases you understand well enough, you don't need a model call at all; you need code. Once you've mapped the edge cases, you write the deterministic pipeline: rules, retrieval, control flow, maybe a small model bolted onto the one genuinely ambiguous step. This is the least glamorous option and the most durable one: reliable, cheap, testable, and not a black box.

Why This Feels Backward (and Why It Isn't)

Andrej Karpathy's "Software 3.0" framing has been everywhere in AI circles since his 2025 "Software Is Changing (Again)" talk: software moved from Software 1.0 (humans hand-write code) to Software 2.0 (humans train neural network weights) to Software 3.0 (humans write natural-language prompts, treating the model itself as a new kind of programmable computer, with everything in its context window acting as the program). At the frontier, that arc is real; natural language keeps unlocking categories of software that used to require a full engineering team.

But zoom into any single feature inside an actual product, and the maturity curve runs the other way. You start at 3.0, a prompt against a frontier model, because that's the fastest way to find out if the idea works at all. Once it works and repeats, you climb down to 2.0: weights you own. Once you fully understand it, you climb down further to 1.0: code you can read.

Both arcs are true at the same time. Karpathy's arc is about what becomes possible. This arc is about what becomes worth hardening, once you've learned the actual shape of the problem. The frontier keeps pushing the ceiling up. Underneath it, mature teams keep pushing their own floor down.

The Receipts

This isn't just a personal theory; it's showing up everywhere once you look for it.

Stanford University's DSPy framework is this pattern turned into an actual engineering discipline. Instead of hand-tuning prompt strings forever, you write a declarative "signature" of what a step should do, and a compiler decides, and re-decides, every time the underlying model or data changes, whether that step should run as a prompt, a set of few-shot examples, or fine-tuned weights. The program is code. The model call becomes just one swappable implementation detail inside it.

Token prices, meanwhile, keep collapsing. One 2026 analysis of pricing across hundreds of models estimated something like a 600x drop in token costs since 2020, with cheaper model tiers now halving in price faster than Moore's Law ever moved. That actually complicates a naive cost argument for fine-tuning low-stakes, high-volume tasks; the API might already be close to free. What fine-tuning and code increasingly buy you isn't just savings; it's control, latency, and moat.

Specialization keeps beating generality on narrow, well-defined tasks. A recent study on structured contract extraction found domain-trained small models matching or beating frontier general-purpose LLMs, at a fraction of the cost and deployable entirely inside enterprise infrastructure. That's tier 2, working exactly as advertised.

And not everyone agrees on the timing, which is worth holding onto rather than smoothing over. Some sharp voices in AI investing argue the opposite case: frontier labs will keep out-improving your custom fine-tune faster than you can maintain it, so unless you're sitting on genuinely proprietary data, the better bet is to keep riding the API and pour your effort into the product wrapped around it. That's a real, unresolved tension. It's exactly why this is a portfolio decision and not a fixed rule.

The Part Nobody's Actually Managing

Here's what I think most roadmaps get wrong: this isn't three sequential stages for your product. 

It's three tiers running simultaneously, for different capabilities, all the time. Your onboarding flow might already be sitting at tier 3 because you nailed it a year ago. Your newest agentic feature is at tier 1 because you shipped it three weeks ago and don't know its failure modes yet. Something in the middle just crossed the volume threshold where fine-tuning finally pays for itself.

That's not a one-time build-vs-buy fork. That's a resource allocation problem, a live one, shifting every quarter as usage patterns, model prices, and your own understanding of the task all move independently of each other. Most AI roadmaps are still built like it's a single decision made once at kickoff.

A few questions I've found useful for figuring out where a given capability actually belongs:

• How often does it run? Low volume, sporadic — stay on the API. The fixed cost of owning it isn't worth paying yet.
• Is "correct" still moving? If your own definition of a good output changed last month, don't freeze it into weights or code. You'll just have to redo the work.
• Could a competitor replicate this with the same API call you're making? If yes, it was never your moat. Don't over-invest in owning it.
• What's your tolerance for a black box? Audit, compliance, and debuggability needs can pull a capability toward code even before the economics demand it.
• Do you actually have the data? You can't responsibly fine-tune or hard-code what you can't yet describe with real, labeled examples.

Where This Leaves Us

Having three ways to solve a problem instead of one is genuine abundance. 

A few years ago, "write the code yourself" was the only option on the table. That's insane!

But abundance isn't free; it converts every roadmap into a standing allocation problem: what stays on the frontier, what gets pulled in-house, what gets frozen into something boring and reliable. Decided over and over, forever, as the ground shifts under all three tiers at once.

Which of your product's capabilities do you think is sitting at the wrong tier right now?