Stop Fine-Tuning Everything: A Decision Framework for Model Adaptation

The default playbook for adapting a foundation model looks like this: grab a pre-trained model, collect labeled data, fine-tune, deploy. It works often enough that teams rarely question it, but fine-tuning is one of six adaptation strategies. Picking the wrong one costs you weeks of engineering time and thousands in compute - sometimes for accuracy you'll never get back.

The real question isn't how to fine-tune. It's whether you should.

I've spent the last several years working on model adaptation across domains where labeled data is scarce and off-the-shelf models fall flat. This article lays out the decision framework I wish I'd had when I started: how to choose the right strategy based on how much labeled data you have, how far your domain sits from the model's pre-training distribution, and how much you can spend.

The Six Strategies

There are six ways to adapt a foundation model to a new task. I'm listing them in order of increasing investment:

1. Prompt Engineering

Modify the input to the model by writing instructions, providing context, and being specific about output format. This needs zero labeled data.

2. Few-Shot/In-Context Learning (ICL)

Include 3–20 labeled examples directly in the prompt. Still no training. The model picks up the pattern from examples at inference time.

3. Retrieval-Augmented Generation (RAG)

Give the model access to your data at inference time by retrieving relevant documents and injecting them into the prompt. No training required. The model stays the same, and gets what it needs to know at inference time.

4. Fine-Tuning

Update model weights on your labeled dataset. Requires hundreds to thousands of labeled examples. The model permanently learns your task. Modern parameter-efficient methods like  LoRA and QLoRA have made this dramatically cheaper — you can fine-tune a 7B-parameter model on a single GPU by updating less than 1% of the weights.

5. Domain-Specific Pre-Training (Self-Supervised Learning)

Take a pre-trained foundation model and continue pre-training it on a large body of unlabeled domain data using self-supervised objectives. You're keeping everything the model already knows — general visual features, language structure, spatial reasoning — and teaching it what your domain looks like on top of that. Then you fine-tune on a much smaller labeled set - because SSL teaches the model what your domain looks like, but not what you want it to do. Self-supervised objectives have no notion of your labels or your task; you still need fine-tuning to map those learned representations to actual decisions. The key here is that you start from a pre-trained model and not from scratch.

6. Train From Scratch

Initialize a model with random weights and train it entirely on your own data. The model starts with knowing nothing. You teach it everything from the ground up. Maximum control, maximum cost. This is rarely justified unless your domain is so unique that no existing pre-trained model carries useful knowledge (novel modalities, proprietary data formats) or regulatory/IP constraints prevent using pre-trained weights.

Most teams jump straight to Strategy 4, and that is costly.

The Decision Framework

Walk through these questions in order, and stop at the first one that meets your bar.

1. Can prompting meet your accuracy bar?

• Yes: Prompt Engineering. Ship it and iterate.
• No: Keep going.

2. Do you have 5–50 high-quality examples?

• Yes: Few-Shot/ICL. Prototype first, commit to training later.
• No: Keep going.

3. Why is the model getting it wrong?

• It doesn't have the right information (e.g., your docs, your product catalog, your policies): RAG. Keep knowledge external and updatable.
• It has the information but doesn't use it correctly (e.g., wrong format, bad reasoning, inconsistent outputs): Keep going.

4. Is your domain close to the model's pre-training data?

• Yes: Fine-Tune (LoRA). Update the weights on your labeled set.
• No: One more question.

5. Do you have large unlabeled domain data?

• Yes: Domain Pre-training (SSL) + Fine-Tune. Teach the model your world, then your task.
• No: Train From Scratch. (Are you sure? Try a different base model first.)

The question most teams skip is the third: knowledge vs. behavior. And the question after that, domain distance, is what separates a quick fine-tune from a multi-week pre-training investment. If your data looks nothing like what the model was pre-trained on, fine-tuning a small labeled set won't close the gap. 

When to Use Which

Prompt Engineering

• Your task is well-defined and the model already "knows" the domain
• You need to ship today, not next month
• Your accuracy bar is 80%, not 95%

Anti-pattern: Teams that spend three weeks prompt-engineering a task that needs fine-tuning. If you're past version 15 of your prompt and still not hitting your bar, stop. You likely need additional signals in your data.

In-Context Learning/Few Shot

• You have 5–50 gold-standard examples
• Your task has clear input-output patterns
• You want to prototype before committing to training

Anti-pattern: Treating ICL as a permanent solution. It works for prototyping, but at scale you're paying inference cost for those examples on every single call. If you're running 10K requests/day with 20-shot prompts, you're burning tokens on examples that could be baked into the weights.

RAG

• The model knows how to do the task but doesn't have the information
• Your knowledge base changes frequently (product catalogs, policies, documentation)
• You need answers grounded in specific sources with citations
• Fine-tuning would bake in facts that go stale

RAG is the strategy most teams overlook when they reach for fine-tuning. If your model is generating plausible but wrong answers about your company's products, the problem isn't the model's behavior but rather the model's knowledge. Fine-tuning on your docs will help temporarily, but the moment those docs change, you'll need to retrain. RAG keeps knowledge external and updatable. Think of this as you knowing accounting, but to help your client, you need some account information and numbers from them.

Anti-pattern: Fine-tuning on your knowledge base to teach the model facts. You're burning compute to memorize information that may be frequently updated. Use RAG for knowledge, fine-tuning for behavior.

Fine-Tuning

• You have 500–10K labeled examples
• Your domain is close to the model's pre-training distribution
• You need consistent quality at scale that you can reproduce

Fine-tuning doesn't always mean updating every weight of the model. Parameter-efficient methods like LoRA (Low-Rank Adaptation) let you fine-tune by updating small adapter matrices. This cuts GPU memory by 60–80% and training time proportionally. If you're fine-tuning in 2025+, you should be using LoRA or QLoRA unless you have a specific reason not to.

Anti-pattern: Fine-tuning on noisy labels. If your labeled data was produced by labelers who disagreed with each other 30% of the time, you're training on noise. A clean set of 500 examples will outperform a noisy 5,000.

Domain-Specific Pre-Training

• Your domain looks different from the checkpoint/model you have chosen as your foundation
• You have lots of unlabeled domain data (10K+ samples)
• Labeled data is expensive or hard to get (medical, legal, scientific, industrial)
• Fine-tuning alone plateaus well below your target accuracy

This is the most underused strategy. People confuse it with training from scratch, but it is different. SSL starts from a pre-trained model with some existing knowledge, and you are leveraging that knowledge to build on top and extend it. For example, if you are working with pathology slides or some sensor data, the pre-trained model has no idea about those domains, but it still has knowledge about edges, shapes, textures, etc that is valuable to leverage. Once you teach the model your world, you can fine-tune on a small labeled set to teach it your specific task. For example, detecting regions of a slide that are cancerous.

Train From Scratch

• You have millions of samples
• Your domain is genuinely unique (novel modality/proprietary data format)
• No pre-trained model carries useful representations for your input type
• You need full control over architecture and training dynamics
• Regulatory or IP constraints prevent using pre-trained weights

When you train from scratch, you start with random weights. The model knows nothing. Every feature —  from low-level patterns up to high-level abstractions — must be learned entirely from your data. With SSL, you inherit all of that for free from the pre-trained model and only teach the domain-specific parts.

That's why the data requirements are so different: SSL needs 10K–100K unlabeled samples to bridge the domain gap; training from scratch needs millions to learn everything a foundation model already knows plus your domain.

Anti-pattern: Training from scratch because "we want to own the model". Unless you have a concrete technical or legal reason or your data is a genuinely novel modality that no pre-trained model has ever seen, you're throwing away billions of tokens of pre-training for no benefit. Try SSL first.

The Cost-Accuracy Tradeoff

Here are some considerations as a reference while you choose which strategy works for your use-case:

The sweet spot for most teams is somewhere between fine-tuning and domain pre-training, but the decision should be driven by the domain gap - how different your data is from what the model has seen already. Collecting more labeled data is not always the solution, and can be counter-productive. A good test for this is to plot the accuracy vs. data size. A plot that plateaus below your accuracy target shows you that more data is not going to bridge the gap.

What the curve tells you:

• Curve still climbing at your largest subset → collect more labels. The model is still learning.
• Early plateau → the bottleneck is representation quality, not label quantity. Try domain pre-training or a different base model.
• High variance between runs at the same size → your labels are noisy or inconsistent. Fix labeling quality before scaling quantity.
• Train accuracy >> validation accuracy (overfitting) → insufficient data diversity. Your labeled set doesn't cover the distribution. Collect more varied examples, not just more examples.

The Bottom Line

The most expensive mistake in model adaptation isn't picking the wrong hyperparameters, but rather picking the wrong strategy entirely. Before you spin up a training job, walk the decision tree. Most of the time, the answer is simpler and cheaper than you think.