We Tested Context7 With ZK Documentation: Here's What We Learned

AI coding assistants have become increasingly capable in understanding not only code but also project-specific documentation. In this experiment, we tested Context7, a new MCP server, to see how well it could work with a large-scale, real-world documentation set — ZK Framework’s developer documentation.

For readers unfamiliar with ZK, it’s a Java-based web framework that allows developers to build rich web applications with minimal JavaScript by providing server-side components. Because ZK documentation covers both UI and server-side concepts, it serves as a good benchmark for evaluating how AI models handle complex, domain-specific technical text.

Our Approach

We ran a controlled experiment to see if the Context7 MCP server could improve AI-generated answers to ZK Framework questions. Our setup was straightforward: using the same AI model (Claude Code) with two conditions — with and without Context7 MCP server — across ten common ZK questions.

Executive Summary

Expected Result

Context7 should help by providing up-to-date documentation.

Actual Result

• Without Context7: 73.8% (PASS)
• With Context7: 59% (FAIL)
• Performance decreased by about 15 percentage points when using Context7

Why This Matters

• Context7 is popular in the AI coding community
• Understanding when it works (and doesn’t) helps developers choose the right tools
• Highlights the importance of how documentation retrieval systems interact with AI models

Background: What Is Context7?

Context7 is a Model Context Protocol (MCP) server by Upstash that automatically retrieves documentation for over 33,000 libraries. Used by AI coding assistants like Cursor and Claude Code, it promises “up-to-date, version-specific documentation.”

How Context7 Works (based on their official documentation):

• Dynamically retrieves documentation from source repositories when triggered
• Detects the specific library mentioned in prompts
• Fetches version-specific documentation and code examples
• Injects relevant documentation directly into the AI model’s context

What Context7 provides:

• Up-to-date code snippets and examples
• Version-specific documentation
• Focused on providing current information to prevent AI hallucinations

Important note: The exact internal architecture (how Context7 parses, indexes, and retrieves documentation) is not publicly documented. Based on community research and reverse engineering efforts (see this Reddit discussion), Context7 appears to use some form of semantic search and snippet extraction, but the specific implementation details remain proprietary.

Critical note: ZK documentation already works well with Claude Code directly. Our test focused specifically on Context7’s retrieval quality.

The Experiment Design

Creating the Test

First, I used Claude Code to create 10 common ZK Framework questions and their correct answers by searching and reading the official ZK documentation directly. This became our answer key.

Control Group (Baseline — Without Context7):

• AI: Claude Code (Claude Sonnet 4.5)
• Documentation: None - answers based on the AI model’s own knowledge
• Method: Claude answers questions without accessing any documentation

Test Group (With Context7):

• AI: Claude Code (same model)
• Documentation: Via Context7 MCP server only
• Method: Context7 retrieves ZK documentation snippets. Claude answers using those snippets combined with its own knowledge
• Note: Claude Code did NOT search ZK documentation directly in this test

The 10 Questions

1. How to upgrade ZK from version 8 to version 10?
2. What is MVVM in ZK, and how do I enable it?
3. How do I display a list of data in a Listbox component?
4. How do I load my own zk-label.properties for internationalization?
5. What is the difference between MVC and MVVM in ZK?
6. What is ID Space in ZK, and how does it work?
7. How do I create a macro component in ZK?
8. How do I enable sorting in a Listbox?
9. How does ZK prevent CSRF (Cross-Site Request Forgery) attacks?
10. How do I handle events between components in ZK?

Evaluation Method

• Use Claude code to give scores based on the answer key from the actual ZK documentation
• Score each answer 0 ~ 100 based on:
  • Technical accuracy
  • Completeness
  • Correctness of examples
  • Actionability

The Results: Context7 Did Not Improve Accuracy 

Overall Scores

Not a single question improved with Context7.

Question-by-Question Results

What This Means for ZK App Developers

Should You Use Context7 With ZK?

For These Questions: ❌ Not Recommended

• How to upgrade ZK?
• How does security work?
• Configuration setup
• Architectural concepts

For These Questions: ✅ Might Help

• Quick syntax lookups
• Component creation examples
• Simple API references

Better Approaches for ZK

1. Use Claude Code directly (without Context7)
   • Currently gives better results
   • Can search the full documentation
   • Understands complete guides
2. Search the ZK documentation yourself
   • Visit the official docs at zkoss.org
   • Use the site search for specific topics
   • Read complete guides for complex topics
3. Ask in the ZK Community
   • Forum
   • Get human expertise for complex questions

The Silver Lining

• This test validates that ZK documentation is comprehensive and reliable
• AI models CAN already answer ZK-related questions well when given direct access to documentation
• The issue lies in how documentation is retrieved, not in its quality

Conclusion

Key message: This experiment wasn’t about showing Context7 in a negative light — it was about understanding how AI tools interact with different types of documentation. Context7 performs well for API-style references, but when it comes to frameworks like ZK that emphasize architecture and conceptual guidance, it did not improve the accuracy.

Takeaway for Community

• Test tools before adopting
• Compare with baseline
• Share findings transparently
• Choose tools that match your needs

What We Learned

• Even popular tools can vary greatly depending on the framework’s documentation style.
• Documentation retrieval and context matter as much as content quality.
• Evaluating tools objectively helps developers choose the right fit for their workflow.

AI-assisted development is becoming a normal part of many developers’ workflows. Understanding when AI tools help — and when they don’t — allows developers to make better decisions, avoid misleading outputs, and use these tools more effectively. At the same time, insights like these also help framework and tool maintainers make their documentation more accessible to both humans and AI systems.  

Note: AI tooling evolves rapidly. The results here reflect the state of Context7 and related systems as of October 2025. Future improvements may significantly change how well these integrations perform. 

Reference

ZK Framework Official Documentation Repository

Discussion

Have you tried using AI tools with framework documentation — like Context7, MCP servers, or IDE assistants? What worked well, and what didn’t?

Share your thoughts in the comments — your experience can help other developers make sense of how AI fits into real-world coding workflows.