Key Considerations in Cross-Model Migration

With the rampant development and release of AI models every few days, ML engineers are expected to conduct comprehensive experiments with different models to choose the best-performing one. However, this is often not a straightforward process — it requires both art and structured methodology.

Modifying the underlying prompts while ensuring best practices is a key challenge that is often not discussed much. Moreover, while it may seem straightforward to simply “swap out” the underlying model and its associated prompt, there are several more nuances to consider — tokenizers, context window sizes, instruction-following abilities, sensitivity to prompt formatting, structured response generation, latency-throughput tradeoff, etc.

Whether it's shifting from OpenAI’s GPT models to Anthropic’s Claude or Google’s Gemini, managing prompts effectively across different model architectures is crucial to maintaining performance, consistency, and efficiency. This article explores the key challenges in evaluating and migrating between various closed-source, state-of-the-art frontier LLMs.

Understanding Model Differences

Each AI model family has its own strengths and limitations. Some key aspects to consider include:

1. Tokenization variations – Different models use different tokenization strategies, impacting the input prompt length and its total associated cost.
2. Context window differences – Most flagship models allow a 128K tokens context window. However, Gemini pushes this further to 1M and 2M tokens.
3. Instruction following – Reasoning models prefer simpler instructions, while chat-style models require clean and explicit instructions. 
4. Formatting preferences  – Some models prefer markdown while others prefer XML tags for formatting.
5. Model response structure – Each model has its own style of generating responses, affecting verbosity and factual accuracy. Some models perform better when allowed to "speak freely", i.e., without adhering to an output structure, while others prefer JSON-like output structures. There is interesting research that shows the interplay between structured response generation and overall model performance.

Case Study: Migrating from OpenAI to Anthropic

Tokenization Variations

All model providers pitch extremely competitive per-token costs. For example, this post shows how the tokenization costs have plummeted for GPT-4 in just one year between 2023 and 2024. However, from an ML practitioner’s viewpoint, making model choices and decisions on purported per-token costs can often be misleading. 

A practical case study presented on the comparison between GPT-4o and Sonnet 3.5 exposes the verbosity of Anthropic models’ tokenizers. In other words, Anthropic tokenizer tends to break down the same text input into a larger number of tokens compared to OpenAI’s tokenizer. 

Context Window Differences

Each model provider is pushing the boundaries to allow longer and longer input text prompts. However, different models may handle different prompt lengths differently. For example, Sonnet-3.5 offers a larger context window up to 200K tokens as compared to the 128K context window of GPT-4. Despite this, it is noticed that OpenAI’s GPT-4 is the most performant in handling contexts up to 32K, whereas Sonnet-3.5's performance declines with an increase in prompts longer than 8K to 16K tokens.

Moreover, there is evidence that different context lengths are treated differently within intra-family models by the LLM, i.e., better performance at short contexts and worse performance at longer contexts for the same task. This means that replacing one model with another (from the same or a different family) may result in unexpected performance deviations.

Formatting Preferences

It is unfortunate that even the current state-of-the-art large language models (LLMs) are highly sensitive to minor prompt formatting. This means that the presence or absence of formatting, in the form of markdown and XML tags, can significantly vary the model's performance on a given task.

Empirical results across multiple studies suggest that OpeAI models prefer markdownified prompts, including sectional delimiters, emphasis, lists, etc., whereas Anthropic models prefer XML tags for delineating different parts of the input prompt. This nuance is commonly known to data scientists, and there is ample discussion on the same in public forums (Has anyone found that using Markdown in the prompt makes a difference? [1], Formatting plain text to markdown [2], Use XML tags to structure your prompts [3]).

For more insights, check out the official best prompt engineering practices released by OpenAI and Anthropic, respectively.  

Model Response Structure

OpenAI GPT-4o models are generally biased towards generating JSON-structured outputs. However, Anthropic models tend to demonstrate equal adherence to the requested JSON or XML schema, as specified in the user prompt.

However, making this decision about imposing or relaxing the structures on models’ outputs is a model-dependent and empirically-driven decision based on the underlying task. During the model migration phase, if you choose to modify the expected output structure, it also means that it will entail slight adjustments in the post-processing of the generated responses.

Conclusion

Migrating prompts across AI model families requires careful planning, testing, and iteration. By understanding the nuances of each model and refining prompts accordingly, developers can ensure a smooth transition while maintaining output quality and efficiency.

ML practitioners must invest in robust evaluation frameworks, maintain documentation of model behaviors, and collaborate closely with product teams to ensure the model outputs align with end-user expectations. Ultimately, standardizing and formalizing the model and prompt migration methodologies will equip teams to future-proof their applications, leverage best-in-class models as they emerge, and deliver more reliable, context-aware, and cost-efficient AI experiences to users.

Resources

1. Has anyone found that using markdown in the prompt makes a difference?
2. Formatting plain text to markdown
3. Use XML tags to structure your prompts