Spring AI on Local LLMs

Why Running It Locally

The usual reason given is privacy, and it’s a real one. A prompt sent to a local model never leaves the machine, which changes the conversation with your security people from a negotiation into a non-event. But three other reasons matter more day to day.

1. Cost goes to zero at the margin. Iterating on a prompt means running it fifty times. Against a hosted API, it's a line item and a rate limit. Against a local model, it’s free.
2. Tests stop depending on the Internet. A build that reaches a hosted endpoint is a build that fails when someone modifies a key or a region has a bad afternoon.
3. You learn where the abstraction leaks. Spring AI presents one ChatClient over every provider. Running a small local model is the fastest way to find out which parts of that promise hold and which parts quietly assume you are talking to a frontier model.

The honest cost is quality and speed. An 8-billion-parameter model on a laptop isn’t a frontier model, and the numbers later in this article are not flattering. That trade is fine for development, prompt iteration, and tests, but it’s a decision to deliberately make for production.

The Starting Point

The initial set-up is the following:

• Java 25 (Temurin)
• Spring Boot 4.1.1
• Spring AI 2.0.1
• Local LLM runtime and model (details in the next section)

Additionally, a freshly generated Spring Boot project, the state you get from start.spring.io before anything interesting has happened. Web MVC, Actuator, DevTools, their test companions and the Spring AI BOM:

<properties>
    <java.version>25</java.version>
    <spring-ai.version>2.0.1</spring-ai.version>
</properties>
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-webmvc</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <scope>runtime</scope>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-webmvc-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>${spring-ai.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

One note on the parent: Spring AI 2.0 is built against Spring Boot 4.x and Spring Framework 7. If you are coming from Spring AI 1.x on Boot 3, this is not a version bump you can do in isolation; package names moved on both sides.

Getting a Model onto the Machine

In this article, Ollama is the choice: it downloads models, manages their lifecycle, and serves them over HTTP on port 11434. Let’s take it step-by-step. 

Install Ollama

The Windows installer is a one-liner in PowerShell:

irm https://ollama.com/install.ps1 | iex
# >>> Downloading Ollama for Windows...
# ######################################## 100.0%
# >>> Installing Ollama...
# >>> Install complete. Run 'ollama' from the command line.

Run Ollama the First Time

ollama run pulls the model if it is missing, loads it, and takes you into an interactive prompt. The daemon keeps serving on port 11434 after you leave it:

ollama run llama3.1
# pulling manifest
# pulling 667b0c1932bc: 100% ▕████████████████████████████████████████▏ 4.9 GB
# pulling 0ba8f0e314b4: 100% ▕████████████████████████████████████████▏  12 KB
# pulling 455f34728c9b: 100% ▕████████████████████████████████████████▏  487 B
# verifying sha256 digest
# writing manifest
# success

The one-line Windows command below confirms the model is present: 

ollama list
# NAME               ID              SIZE      MODIFIED
# llama3.1:latest    46e0c10c039e    4.9 GB    12 days ago

The detail that matters for Spring AI is that Ollama serves two APIs at once. Its native API lives under /api and an OpenAI-compatible one lives under /v1. Prove the second is alive before you write any Java, as it saves you from debugging a Spring context when the problem is a daemon: 

curl http://localhost:11434/v1/models
# {"object":"list","data":[{"id":"llama3.1:latest","object":"model","created":1788341287,"owned_by":"library"}]}

Two Ways to Wire It Up

Spring AI ships both a dedicated Ollama starter and an OpenAI one that will successfully talk to anything speaking the OpenAI protocol. Both work against the same daemon. Putting both on the classpath and choosing between them with a property is a good way to feel the difference: 

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

Two starters on the classpath would normally mean two competing ChatModel beans. Spring AI resolves that with spring.ai.model.chat property, which names the provider that is currently chosen. The other starter’s auto-configuration backs off. 

# Specifies the used starter
spring.ai.model.chat=ollama
# Native Ollama starter (active when spring.ai.model.chat=ollama)
spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.model=llama3.1
spring.ai.ollama.chat.temperature=0.7
# OpenAI-compatible route (active when spring.ai.model.chat=openai)
# Ollama does not check the API key, but the starter requires one to be set.
spring.ai.openai.api-key=ollama
# The /v1 must be part of the base-url: the underlying openai-java client
# appends only /chat/completions, so omitting it yields a 404.
spring.ai.openai.base-url=http://localhost:11434/v1
spring.ai.openai.chat.model=llama3.1
spring.ai.openai.chat.temperature=0.7

A note is worth making on spring.ai.openai.api-key=ollama. Ollama ignores the value entirely, but the starter refuses to configure itself without one, so it needs a placeholder. Any non-empty string does the job.

Another aspect worth noting is the base URL needs its own /v1. 

 The natural thing to write is spring.ai.openai.base-url=http://localhost:11434, on the assumption that the client appends the full /v1/chat/completions path. It does not. Spring AI 2.0 delegates to the official openai-java SDK (4.49.0 here), which appends only /chat/completions. Get this wrong and every call comes back as: 

com.openai.errors.NotFoundException: 404: 404

No hostname, no path, no hint, just the status code twice. The fix is to make /v1 part of the base URL, as above. The same applies to any other OpenAI-compatible server you point this at. 

Flipping to the OpenAI-compatible route needs no code change at all: 

./mvnw spring-boot:run -Dspring-boot.run.arguments=--spring.ai.model.chat=openai

Both spring.ai.ollama.chat.temperature and spring.ai.ollama.chat.options.temperature are valid, although the latter is deprecated. The starter’s configuration metadata registers both spellings, so your IDE will not flag either. Pick one and be consistent, because a typo in this family of properties fails silently rather than at startup. 

One Client, Built Once

With a provider configured, Spring AI auto-configures a ChatClient.Builder. The temptation is to build a client inside the controller. Resist it; the client carries defaults that belong to the application, not to one endpoint and ChatClient is thread-safe and meant to be built once and shared. So it becomes a bean:

@Configuration
public class ChatConfig {
  
    @Bean
    public ChatMemory chatMemory() {
        return MessageWindowChatMemory.builder()
                .maxMessages(10)
                .build();
    }
  
    @Bean
    public ChatClient chatClient(ChatClient.Builder builder,
                                 @Value("classpath:/prompts/system-prompt.st") Resource systemPrompt,
                                 ChatMemory chatMemory) {
        return builder
                .defaultSystem(systemPrompt)
                .defaultAdvisors(new SimpleLoggerAdvisor(Ordered.LOWEST_PRECEDENCE),
                        MessageChatMemoryAdvisor.builder(chatMemory).build())
                .build();
    }
}

Three decisions are packed into those twenty lines. 

Prompts Live in Files

defaultSystem() accepts a Resource, so the system prompt is a StringTemplate file under src/main/resources/prompts rather than a string literal in Java. It reads better, diffs better, and can be edited without recompiling: 

You are a generalist. 
Answer as briefly and precisely as you can both general and specific questions.

A system prompt is worth setting even for a toy. Small local models are chatty by default, and a brief system message is the difference between a sentence and five paragraphs of preamble. 

Memory Is an Advisor

Every ChatClient call is stateless unless something makes it otherwise. MessageChatMemoryAdvisor is that something. Before each call, it loads the history for a conversation and appends it to the prompt, and after each call it stores the new user and assistant messages. The store behind it here is MessageWindowChatMemory, an in-memory map keyed by conversation id that keeps the last ten messages and drops the oldest when the window fills. Which conversation a request belongs to is decided per call, in the next chapter.

Logging Sits Closest to the Model

 SimpleLoggerAdvisor logs the request and the response at DEBUG. Its position in the chain is set by the order argument and Ordered.LOWEST_PRECEDENCE puts it last, after the memory advisor has done its work, so what it logs is the prompt the model actually receives, history and format instructions included. It stays quiet until you switch its logger on: 

logging.level.org.springframework.ai.chat.client.advisor=DEBUG

That one line is the most useful debugging aid in this article. When a model misbehaves, the first question is always “what did it actually see?” and this answers it. 

The Endpoint

The controller is thin. It resolves a conversation ID from an optional header, delegates to a service, and echoes the ID back so the caller can continue the same conversation on the next request: 

@RestController
@RequestMapping("/api")
class ChatController {
    private final ChatService chatService;
  
    public ChatController(ChatService chatService) {
        this.chatService = chatService;
    }
  
    @GetMapping("/v{version}/chat")
    public ResponseEntity<Response> chat(@RequestParam String message,
                                         @RequestHeader(value = "X-Conversation-Id", required = false) String conversationId) {
        String id = conversationId == null ? UUID.randomUUID().toString() : conversationId;
        var response = chatService.respond(message, id);
        return ResponseEntity.ok()
                .header("X-Conversation-Id", id)
                .body(response);
    }
}

The v{version} segment is not a hand-rolled convention. Spring Framework 7 has first-class API versioning and Boot 4 configures it from properties: 

spring.mvc.apiversion.use.path-segment=1
spring.mvc.apiversion.supported=1.0
spring.mvc.apiversion.default=1.0

The first line tells MVC that the version is the path segment at index 1, in /api/v1/chat that is v1, and the default parser understands the leading v. The second lists what the application supports, so a request for /api/v2/chat is rejected before any controller runs. The third is the version assumed when a request carries none.

The service is where the ChatClient fluent API earns its keep. One chained call covers the prompt template, the conversation, the tools and the output type:

@Service
public class ChatService {
  
    private final ChatClient chatClient;
    private final Resource userPrompt;
    private final TenantService tenantService;
  
    public ChatService(ChatClient chatClient,
                       @Value("classpath:/prompts/user-prompt.st") Resource userPrompt,
                       TenantService tenantService) {
        this.chatClient = chatClient;
        this.userPrompt = userPrompt;
        this.tenantService = tenantService;
    }
  
    public Response respond(String message, String conversationId) {
        return chatClient.prompt()
                .user(userSpec ->
                        userSpec.text(userPrompt)
                                .param("message", message))
                .advisors(advisorSpec ->
                        advisorSpec.param(ChatMemory.CONVERSATION_ID, conversationId))
                .tools(tenantService)
                .call()
                .entity(Response.class);
    }
}

Line by line, the user message is a template with a {message} placeholder, filled with .param(), which keeps user input out of the template’s structure and the template itself readable: 

Provide a short, to-the-point answer to the below message:

{message}

The advisors() block hands the conversation id to the previously mentioned memory advisor, under the well-known key ChatMemory.CONVERSATION_ID. Two requests with the same X-Conversation-Id header share history. A request without the header starts a fresh conversation and is told its id in the response. tools() and entity() get their own chapters.

GET /api/v1/chat?message=In one sentence, what is quantization of an LLM?

LLAMA3.1: 
Quantization of a large language model (LLM) refers to the process of approximating its weights and 
activations with lower-precision data types (such as integers or fixed-point numbers) to reduce memory 
usage and computational overhead, while minimizing the impact on model performance.

 If you are running with DevTools and edit application.properties or a prompt file under src/main/resources, nothing happens until the file is copied into target/classes. DevTools watches the build output, not the source tree. A rebuild in your IDE triggers it; editing the file in a plain text editor doesn’t, and you’ll spend a while wondering why your fixed base URL is still returning 404.

Answers as Java Records

Text is informative, but typed objects are the reason to put a model behind a Spring service. entity() takes a class, derives a JSON schema from it, appends format instructions to the prompt, and parses what comes back. The endpoint returns this:

public record Response(
        @JsonPropertyDescription("""
                The category of the request message. \
                SPECIFIC when the message refers to a particular tenant, identified by its unique identifier, \
                so that answering requires the tenant information tool. \
                GENERAL for any other message, such as general knowledge or technical questions \
                that can be answered without tenant data.""")
        ResponseCategory category,
        @JsonPropertyDescription("The actual response message text")
        String text) {
}
  
public enum ResponseCategory {
    GENERAL,
    SPECIFIC
}

The @JsonPropertyDescription annotations are not decoration. They flow into the generated schema and the schema is what the model reads, so a description is the only place a field’s meaning gets explained. The enum becomes an enum constraint in the schema, which is why the model reliably answers with one of the two constants rather than a synonym. The category description is deliberately a paragraph, not a label. 

Giving the Model a Tool

A model knows nothing about your data specifics (tenants’ information as part of this experiment). Tool calling is how it asks. You describe a method; the model decides whether to call it and with what arguments, Spring AI executes it and feeds the result back, and the model writes its answer from that. The description is the interface; it’s all the model sees:

@Service
public class TenantService {
  
    private static final Logger logger = LoggerFactory.getLogger(TenantService.class);
  
    private final Map<String, String> tenantInfo = Map.of(
            "TNG", "Tenant TNG is a well known TEM services provider.",
            "FED", "Tenant FED is Roger Federer's foundation.",
            "DJO", "Tenant DJO is Novak Djokovic's foundation."
    );
  
    @Tool(name = "get-tenant-info-by-id",
            description = "Retrieve additional information by tenant id")
    public String infoByTenantId(@ToolParam(description = "The unique identifier of a tenant") String tenantId) {
        logger.info("Retrieving additional info for tenant with id {}", tenantId);
        return tenantId == null ? "N/A" : tenantInfo.getOrDefault(tenantId, "N/A");
    }
}

Passing the bean to .tools(tenantService) in the service is all the registration there is. Spring AI scans it for @Tool methods and builds the JSON schema for their parameters from the @ToolParam descriptions. The log line is deliberate. Whether the model decided to call the tool and what it passed, is otherwise invisible. 

The null guard matters too. A model that decides to call the tool but fails to extract an argument sends nothing for tenantId and a Map.getOrDefault(null, …) on an immutable map throws. Returning "N/A" instead lets the model explain the gap in its own words, which is exactly what happens in the first test case below. 

Testing Without a Model

A local model makes tests possible offline, but it does not make them good tests. The output varies run to run, and every assertion becomes a guess about phrasing. For anything that checks your code rather than the model’s, replace the ChatModel bean with a mock. Spring AI‘s auto-configuration builds the ChatClient.Builder from whatever ChatModel is in the context, so a @MockitoBean is enough to take the daemon out of the picture entirely: 

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.MOCK)
@AutoConfigureMockMvc
class ChatControllerTests {
  
    @MockitoBean
    private ChatModel chatModel;
  
    @Autowired
    private MockMvcTester mvc;
  
    @BeforeEach
    void init() {
        // ChatClient copies the model's defaults into every request, so a bare mock does not work
        when(chatModel.getOptions())
                .thenReturn(ChatOptions.builder().build());
    }
  
    @Test
    void chat() {
        var message = "Local models keep your prompts on your own machine.";
        var modelOutput = """
            {"category": "GENERAL", "text": "%s"}
        """.formatted(message);
        var response = new ChatResponse(List.of(new Generation(new AssistantMessage(modelOutput))));
        when(chatModel.call(any(Prompt.class)))
                .thenReturn(response);
      
        assertThat(mvc.get()
                    .uri("/api/v1/chat")
                        .param("message", "Why run locally?"))
                .hasStatusOk()
                .bodyJson()
                .convertTo(Response.class)
                .isEqualTo(new Response(ResponseCategory.GENERAL, message));
    }
}

The mock returns the JSON a model would return, and the test asserts that your controller, service, template and record mapping serve it back as a Response. The model’s ability to produce that JSON is a separate question and not one a unit test should be asking.

Two things are quietly bypassed here, and it is worth knowing which. Tool execution happens inside the concrete ChatModel implementation, so a mocked model never calls TenantService. The tool is registered but inert. The memory advisor, on the other hand, is real; it runs, stores the exchange in the in-memory window under a freshly generated conversation ID, and does no harm. 

A note on the chatModel mock is worth making. If only mocking ChatModel and stubbing just call(), the first request fails with: 

NullPointerException: Cannot invoke "ChatOptions.mutate()" because the return value of "ChatModel.getOptions()" is null

ChatClient reads the model’s default options and merges them into every request. Mockito returns null for the unstubbed getter and the merge breaks. One line in a @BeforeEach fixes it, but the stack trace points into Spring AI internals rather than at your test, so it costs more time than it should.

Testing the Integration

 With the application started, four requests on the same conversation exercise the whole chain: template, memory, tool, structured output. The X-Conversation-Id header ties them together. What each one did is known from the log line in TenantService and from the advisor log previously switched on. 

Use Case 1: Ask by “id”

curl -G http://localhost:8080/api/v1/chat -H "X-Conversation-Id: 123-1bc-981"
    --data-urlencode "message=Provide info for tenant with id FED"

# {"category":"GENERAL","text":"Tenant information not available"}

The tool was invoked, but with no tenantId. The model recognized that the question needed the tool and then failed to map "FED" onto the parameter, so the method returned N/A and the model paraphrased it.

Use Case 2: Same Question, Phrased With the Parameter’s Own Description

curl -G http://localhost:8080/api/v1/chat -H "X-Conversation-Id: 123-1bc-981"
    --data-urlencode "message=Provide info for tenant with unique identifier FED"

# {"category":"GENERAL","text":"Tenant FED is Roger Federer's foundation."}

Tool invoked, tenantId mapped to FED, correct answer. The only change was “id” becoming “unique identifier,” the exact words in the @ToolParam description. 

Use Case 3. The Winning Phrasing, Different Tenant

curl -G http://localhost:8080/api/v1/chat -H "X-Conversation-Id: 123-1bc-981"
    --data-urlencode "message=Provide info for tenant with unique identifier DJO"
    
# {"category":"GENERAL","text":"Tenant DJO is an abbreviation for DJ Okaya."}

The tool was not invoked. The model answered from nothing, fluently and wrongly. 

Use Case 4: The Identical Request, Sent Again

curl -G http://localhost:8080/api/v1/chat -H "X-Conversation-Id: 123-1bc-981"
    --data-urlencode "message=Provide info for tenant with unique identifier DJO"
    
# {"category":"GENERAL","text":"Tenant DJO is Novak Djokovic's foundation."}

This time the tool was invoked, tenantId mapped to DJO, and the answer is right. Nothing changed between use cases 3 and 4 except the conversation history being one exchange longer. 

Four requests, one tool, three different behaviors: called without arguments, called correctly, and not called at all, the last two for the same input. The temperature of 0.7 explains some of the variance, but the deeper issue is that an 8-billion-parameter model is weak at the two judgment calls tool use requires: should I call this, and what goes in the parameter. 

Two mitigations are visible in the results. Wording the user’s request in the same vocabulary as the @ToolParam description made the difference between use cases 1 and 2, so descriptions should use the words users will use. And the null guard in TenantService turned a would-be exception into a polite non-answer. Neither makes use case 3 go away, that probably needs a lower temperature, a system prompt that instructs the model to always use the tool for tenant questions, or a bigger model.

What It Actually Feels Like

Measured on a laptop, Intel Core Ultra 7 155H, 32 GB RAM, RTX 500 Ada, running llama3.1 8B through the endpoint above:

• 14.2s – first request after startup, three-bullet answer
• 11.3s – same request, model already warm

Double-digit seconds for a paragraph and a tool call adds a second round trip to the model on top. That’s workable for development and unacceptable for a request a user is waiting on, which is the honest headline: local models are a development and testing tool at this size.

The quality gap is sharper than the speed gap, and it’s worth seeing rather than being told about. Asked a question about the very tool serving it:

GET /api/v1/chat?message=In one sentence: what is Ollama?

LLAMA3.1:
Ollama is a brand of vegan, plant-based egg substitute made from a combination of pea protein, 
potato starch and other ingredients.

Confident, fluent, well-formed and entirely invented. The model’s training data predates the tool it’s running inside. Put that next to "DJ Okaya" from before and the pattern is clear. This is the most useful thing about developing against a small local model: the failure modes you will eventually have to design around are loud here instead of rare. Whatever guardrails, validation, and grounding your application needs in production, you will notice you need them on day one. 

Resources

[1] – POC code is here

[2] – The picture was taken in Zakynthos, Greece