Building a Rust Command Line Interface to Chat With Llama 3.2

As a developer learning Rust, I wanted to build a practical project to apply my new skills. With the rise of large language models like Anthropic's Llama 3.2, I thought it would be interesting to create a Rust command line interface (CLI) to interact with the model.

In just a couple of minutes, I was able to put together a working CLI using the Ollama Rust library. The CLI, which I call "Jarvis," allows you to chat with Llama 3.2, as well as perform some basic commands like checking the time, date, and listing directory contents.

In this post, I'll walk through the key components of the Jarvis CLI and explain how you can use Rust to interface with Llama 3.2 or other large language models. By the end, you'll see how Rust's performance and expressiveness make it a great choice for AI applications.

The Jarvis CLI Structure 

The main components of the Jarvis CLI include:

1. JarvisConfig struct

• Defines the available commands
• Methods to validate commands and print help text

2. Command Handling Logic in main()

• Parses command line arguments
• Invokes the appropriate function based on the command

3. Functions for Each Command

• time - Gets current time
• date - Gets today's date
• hello - Prints a customizable greeting
• ls - Lists directory contents
• chat - Interacts with Llama 3.2 using Ollama lib

Here's a condensed version of the code:

struct JarvisConfig {
    commands: Vec<&'static str>,
}

impl JarvisConfig {
    fn new() -> Self {...} 
    fn print_help(&self) {...}
    fn is_valid_command(&self, command: &str) -> bool {...}
}

#[tokio::main]
async fn main() {
    let config = JarvisConfig::new();
    let args: Vec<String> = env::args().collect();
    
    match args[1].as_str() {
        "time" => {...}
        "date" => {...}
        "hello" => {...}
        "ls" => {...}
        "chat" => {
            let ollama = Ollama::default(); 
            
            match ollama
                .generate(GenerationRequest::new(
                    "llama3.2".to_string(),
                    args[2].to_string(),
                ))
                .await
            {
                Ok(res) => println!("{}", res.response),
                Err(e) => println!("Failed to generate response: {}", e),
            }
        }
        _ => {
            println!("Unknown command: {}", args[1]);
            config.print_help();
        }
    }
}

Using Ollama to Chat with Llama 3.2 The most interesting part is the "chat" command, which interfaces with Llama 3.2 using the Ollama Rust library.

After adding the Ollama dependency to Cargo.toml, using it is fairly straightforward:

1. Create an Ollama instance with default settings:

let ollama = Ollama::default();

2. Prepare a GenerationRequest with the model name and prompt:

GenerationRequest::new(
    "llama3.2".to_string(), 
    args[2].to_string()
)

3. Asynchronously send the request using ollama.generate():

match ollama.generate(...).await {
    Ok(res) => println!("{}", res.response),
    Err(e) => println!("Failed to generate response: {}", e),
}

That's it! With just a few lines of code, we can send prompts to Llama 3.2 and receive generated responses.

Example Usage 

Here are some sample interactions with the Jarvis CLI:

$ jarvis hello  
Hello, World!

$ jarvis hello Alice
Hello, Alice! 

$ jarvis time
Current time in format (HH:mm:ss): 14:30:15

$ jarvis ls /documents
/documents/report.pdf: file
/documents/images: directory

$ jarvis chat "What is the capital of France?" 
Paris is the capital and most populous city of France.

While Python remains the go-to for AI/ML, Rust is a compelling alternative where maximum performance, concurrency, and/or safety are needed. It's exciting to see Rust increasingly adopted in this space!

Conclusion

In this post, we saw how to build a Rust CLI to interact with Llama 3.2 using the Ollama library. With some basic Rust knowledge, we could put together a useful AI-powered tool in just a couple of minutes. Rust's unique advantages make it well-suited for AI/ML systems development. As the ecosystem matures, I expect we'll see even more adoption.

I encourage you to try out Rust for your next AI project, whether it's a simple CLI like this or a more complex system. The performance, safety, and expressiveness may surprise you!