Docs

Docs

LLM Providers

Connect the AIOrchestrator to Spring AI, LangChain4j, or a custom LLM framework using the LLMProvider interface.

An AI framework — such as Spring AI or LangChain4j — is a Java library that handles the protocol details of talking to LLM services like OpenAI or Anthropic. The orchestrator plugs into your chosen framework through the LLMProvider interface. Create a provider by instantiating the appropriate implementation directly. Two implementations are provided: one for Spring AI and one for LangChain4j.

Important
Memory Window Limit
Both built-in providers maintain a 30-message memory window. Older messages are evicted from the provider’s working memory. The orchestrator’s getHistory() retains the full conversation, but the LLM only sees the most recent 30 messages.

Spring AI

SpringAILLMProvider supports both streaming and synchronous Spring AI models.

Source code
Java
// From ChatModel - use an implementation of Spring AI ChatModel
ChatModel chatModel = OpenAiChatModel.builder()
    .openAiClient(...).options(...).build();
SpringAILLMProvider provider = new SpringAILLMProvider(chatModel);

// From ChatClient - use a Spring AI ChatClient
ChatClient chatClient = ChatClient.builder(...)
    .defaultAdvisors(...).build();
SpringAILLMProvider provider = new SpringAILLMProvider(chatClient);

When created from a ChatModel, the provider manages its own conversation memory using a 30-message window. When created from a ChatClient, memory must be configured externally on the client.

Streaming is enabled by default. To disable it, call setStreaming(false):

Source code
Java
provider.setStreaming(false);

In synchronous mode, the whole exchange runs in the request that triggered it and blocks the UI until the response is complete. See Background Execution for keeping the UI responsive during long prompts.

Note
History Restoration with ChatClient
A provider created from a ChatModel restores the conversation into its own memory, so withHistory() and reconnect() need no extra work. A provider created from a ChatClient cannot do that — the application owns that client’s memory — so its setHistory() does nothing beyond logging what it observed: a warning when the client carries no chat memory advisor or no default conversation id, since a restored conversation then never reaches the LLM. Load the conversation into the client’s own ChatMemory before passing the client to the provider, or use new SpringAILLMProvider(chatModel) and let the provider handle it. The orchestrator’s own conversation history and the Message List are restored either way.

LangChain4j

LangChain4JLLMProvider supports both streaming and synchronous LangChain4j models. The mode is determined by the model type passed to the constructor:

Source code
Java
// Streaming mode - use an implementation of LangChain4j StreamingChatModel
StreamingChatModel streamingChatModel = OpenAiStreamingChatModel.builder()
    .apiKey(...).modelName(...).build();
LangChain4JLLMProvider provider = new LangChain4JLLMProvider(streamingChatModel);

// Synchronous mode - use an implementation of LangChain4j ChatModel
ChatModel chatModel = OpenAiChatModel.builder()
    .apiKey(...).modelName(...).build();
LangChain4JLLMProvider provider = new LangChain4JLLMProvider(chatModel);

The provider manages its own conversation memory using a 30-message window.

Synchronous mode blocks the UI for the duration of each exchange; see Background Execution.

Framework Features

An LLM provider is a thin adapter: it hands each prompt to the framework’s own client and streams back what that client produces. Everything else the model needs is set up on the client, in the framework, before the provider is created. The orchestrator works with whatever the client has been configured with and adds no framework features of its own.

Model Context Protocol (MCP) servers are the most common example. Neither the orchestrator nor the provider is an MCP client. Configure Spring AI’s MCP client — its transport, its credentials, and which of the server’s tools the model may see — register the resulting tool callbacks on a ChatClient, and wrap that client in a provider:

Source code
Java
ChatClient chatClient = ChatClient.builder(chatModel)
        // Tool callbacks from Spring AI's MCP client
        .defaultTools(mcpToolCallbacks)
        .build();
SpringAILLMProvider provider = new SpringAILLMProvider(chatClient);

The MCP tools then reach the model on every turn, next to any tools registered on the orchestrator through withTools() or a controller. The model calls all of them the same way, so tool names have to be unique across every source and match ^[a-zA-Z0-9_-]{1,64}$ — validated at build time for controller tools, but not for names that come from an MCP server. Give the client a chat memory advisor and a default conversation id as well, since a provider created from a ChatClient leaves conversation memory to the application.

Retrieval-augmented generation against a vector store, guardrails and moderation, observability, model options such as temperature and token limits, and structured output with schema validation all work the same way: configure them on the framework’s client, then wrap it in a provider. The framework’s own reference documentation is where to look for the details:

Tip
The System Prompt Comes From the Orchestrator
The orchestrator sends its own system prompt on every turn, which replaces any defaultSystem() text configured on the client. Put instructions for the model — including when to reach for the client’s tools — in the system prompt passed to AIOrchestrator.builder(), not in the client’s defaults.
Note
MCP With LangChain4j
LangChain4JLLMProvider is built from a ChatModel or StreamingChatModel and runs the tool-calling loop itself, so LangChain4j’s McpToolProvider — which attaches to an AI service built with AiServices — has nowhere to plug in. With LangChain4j, MCP tools currently require a custom LLM provider built around an AI service, which then gets the tool provider, memory, and tool loop from LangChain4j. With Spring AI, the ChatClient setup above is all that’s needed.

Tool Call Limits

A turn that uses tools is a loop: the model asks for tool calls, the provider runs them and calls the model again with the results, until the model answers. Both built-in providers bound that loop per turn. By default, the model may call any single tool at most 40 times and all tools together at most 150 times in one turn. Without a bound, a request the model can’t satisfy keeps calling the model and the tools until the application is stopped.

When a limit is exceeded, the turn fails with a ToolCallLimitExceededException, whichever provider runs it. The Message List shows its generic error message, and the exception reaches the response listener and AIController.onResponse() as the error of the turn. Its message names the limit that was exceeded and, for a per-tool limit, the tool:

Source code
Java
.withResponseListener(event -> {
    event.getError()
            .filter(ToolCallLimitExceededException.class::isInstance)
            .ifPresent(error -> log.warn(
                    "Turn stopped: {}", error.getMessage()));
})

LangChain4j

LangChain4JLLMProvider runs the loop itself and enforces the limits. Change them with setMaxCallsPerTool() and setMaxTotalToolCalls(). A value of 0 removes that limit. The values are read when a turn starts, so a change applies from the next prompt.

Source code
Java
LangChain4JLLMProvider provider = new LangChain4JLLMProvider(chatModel);
provider.setMaxCallsPerTool(10);
provider.setMaxTotalToolCalls(0); // No limit across all tools

The provider checks the limits before running the tool calls of a response. When a call would take a count past its limit, none of the tool calls in that response run, and the model isn’t called again. The failed turn leaves only its prompt in the provider’s memory: tool calls and their results are sent to the model only within the turn they belong to and are never kept between turns, so the next prompt continues from where the last completed turn left the conversation.

Spring AI

SpringAILLMProvider adds no limit of its own. Spring AI bounds the loop with the same defaults, 40 calls per tool and 150 in total per turn, with either constructor of the provider. When Spring AI stops the loop, the provider fails the turn with the ToolCallLimitExceededException described above, carrying Spring AI’s own message about the limit. That message may remain in the chat memory, with either constructor: the provider doesn’t rewrite what Spring AI’s advisors stored.

The limits belong to the ChatClient, so tune or remove them there and pass the client to the ChatClient constructor of the provider; the ChatModel constructor builds a client with Spring AI’s defaults. How to configure them is described under Tool Call Limits in the Spring AI documentation.

A custom provider decides itself whether, and how, it bounds its loop.

Background Execution

A synchronous provider produces the response on the thread that asks for it. For a prompt sent from the browser, that is the request thread: the request does not return until the model has produced the complete response, including any tool calls along the way. The interface freezes for the whole wait, and since the request holds the session lock, other views in the same session wait too.

Background execution moves the exchange to a background thread instead. Enable it on either built-in provider:

Source code
Java
provider.setBackgroundExecution(true);

The user’s message and an empty assistant message then appear immediately, the UI stays responsive, and the response is filled in when the model finishes. The setting is off by default. It’s read for each prompt, so it can be changed at any time; the next prompt uses the new mode. It has no effect with a streaming model, whose response already arrives on the LLM client’s own threads.

The response is now produced outside any request, so it reaches the browser through server push or polling. Enable push by annotating the application shell with @Push (see Server Push), or enable polling with UI.setPollInterval(). Without either, the response only shows up with the next request the browser happens to make — the page looks stuck even though the turn completed on the server. The provider logs a warning, once per provider instance, when neither is active. Manual push mode is not enough on its own, because nothing in the framework calls ui.push() for the application.

Everything that happens before the model is called still runs in the request thread: the request interceptor, adding the user’s message and the empty assistant message to the Message List, AIController.onRequest(), the request listener, and the session context supplier. The model calls, every tool execution, and the ResponseListener run on the background thread, where UI.getCurrent() and other Vaadin thread locals return null and components must not be touched directly. Thread-bound framework state, such as Spring Security’s SecurityContext, is absent there for the same reason.

Wrap component access in ui.access(), or capture what a tool needs in AIController.onRequest() while the request thread is still current — see Tool Calling & Programmatic Prompts and Controllers. The built-in controllers already handle this. AIController.onResponse() is the exception: the orchestrator calls it through ui.access(), so it can update components directly.

Note
One Prompt at a Time
The orchestrator processes one prompt at a time. In the default synchronous mode, a message submitted while a turn is running waits for the session lock and is processed when the turn ends. With background execution the lock is free, so the same message is rejected and dropped with a server-side warning — and the Message Input has already cleared its text.

If the user closes or reloads the browser tab while a turn is running, the turn still completes on the server: the response is recorded in the conversation history and the ResponseListener fires as usual. Only the UI updates are skipped, along with AIController.onResponse(), which needs an attached UI. The setting itself lives on the provider and is not serialized with the session — apply it again to the recreated provider after a session restore, before passing it to reconnect(). See Conversation History & Session Persistence.

Custom LLM Providers

Implement the LLMProvider interface to connect to any LLM framework:

Source code
Java
public class MyLLMProvider implements LLMProvider {

    @Override
    public Flux<String> stream(LLMRequest request) {
        // Return a reactive stream of response tokens
        // request.userMessage()  -- the user's prompt
        // request.attachments()  -- any file attachments
        // request.systemPrompt() -- the system prompt
        // request.tools()        -- registered tool objects
        // request.metadataSink() -- consumer for response metadata
    }

    @Override
    public void setHistory(List<ChatMessage> history,
            Map<String, List<AIAttachment>> attachmentsByMessageId) {
        // Restore conversation context
    }
}

The response stream carries text only. The provider can also publish the finish reason and token usage of the turn through the metadataSink() consumer on the request. Each call carries everything observed so far and replaces the value of any earlier call, so publish whenever the provider learns more — a turn that fails midway has then still reported what was observed. Pass null for any value the framework doesn’t report; a provider that observes no metadata never calls the consumer. See Response Metadata for how applications read it.

The orchestrator calls stream() on the thread that triggers the prompt and subscribes to the returned stream on that same thread — whether a turn runs in the background is decided entirely by the implementation. An implementation whose LLM call blocks should schedule that call itself; otherwise it occupies the request thread and holds the session lock for the whole turn. See Background Execution for how the built-in providers expose this as a setting.

Updated –