LM - DSRs

The LM struct is the provider client: a thin wrapper over OpenAI-compatible APIs with built-in retries, optional response caching, and history tracking. You rarely call it directly; a predictor uses an adapter to format a signature and send the result through the configured LM, which keeps business logic separate from transport. Configure one globally with configure(lm), or attach one per predictor with PredictBuilder::lm(...).

use dspy_rs::{init_tracing, LM};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    init_tracing()?;

    // OpenAI - API key automatically read from OPENAI_API_KEY env var
    let lm = LM::builder()
        .model("gpt-4o-mini".to_string())
        .temperature(0.7)
        .max_tokens(512)
        .build()
        .await?;

    // Or explicitly provide API key
    let lm = LM::builder()
        .model("gpt-4o-mini".to_string())
        .api_key("your-api-key".into())
        .build()
        .await?;

    Ok(())
}

Responsibilities

LM handles three core responsibilities:

Configuration - Stores provider credentials, model selection, and inference parameters (eg: temperature)
API Execution - Takes pre-formatted Chat messages and executes HTTP calls to the LLM provider
Response Caching - Optionally stores input/output pairs to avoid duplicate API calls

Structure

LM is built using the builder pattern. The builder collects an LMConfig, the serializable data half, and build() initializes the live client. The config holds:

model - Model identifier (e.g., “gpt-4o-mini” or “openai:gpt-4o-mini”)
api_key - Provider API credentials (optional for local servers)
base_url - API endpoint URL (optional, inferred from model provider)
temperature - Sampling temperature (default: 0.7)
max_tokens - Maximum completion tokens (default: 512)
max_tool_iterations - Upper bound on tool-loop round trips (default: 10)
max_retries - Additional attempts after a transient failure (default: 2)
retry_base_delay_ms - Base delay for exponential retry backoff (default: 250)
cache - Enable response caching (default: false)

The live LM adds:

client - Internal provider client (initialized during build)
cache_handler - Optional response cache (initialized during build if enabled)

Cloning an LM is cheap - clones share the same HTTP client and cache via Arc, making them ideal for concurrent use.

Construction and configuration

The LM::builder() must be awaited with .build().await because client initialization is async.

Local server usage

For local OpenAI-compatible servers (vLLM, Ollama, etc.), provide base_url without an api_key:

let lm = LM::builder()
    .base_url("http://localhost:11434".to_string())
    .model("llama3".to_string())
    .build()
    .await?;

Custom OpenAI-compatible endpoints

For custom endpoints requiring authentication, provide both base_url and api_key:

let lm = LM::builder()
    .base_url("https://my-custom-api.com/v1".to_string())
    .api_key(my_api_key.into())
    .model("custom-model".to_string())
    .build()
    .await?;

Clone semantics: LM implements Clone; clones share the underlying client and cache via Arc, so they see the same history while carrying their own config copy.

API Reference

You can browse the full LM module reference on docs.rs.

Global vs explicit usage

Global: configure(lm) sets the process-wide default LM used by predictors.
Per-instance override: Attach an LM to a specific predictor with PredictBuilder::lm(...), which bypasses the global; or build a second LM and call configure(lm) before the specific call.

Async execution and sync entry

Async: LM building and calls are async; prefer using an async runtime (Tokio).
Sync-style: If you need a plain fn main, create a runtime and block_on the async work.

Async (Tokio)
Sync

use dspy_rs::{init_tracing, LM};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    init_tracing()?;

    let lm = LM::builder()
        .model("gpt-4o-mini".to_string())
        .build()
        .await?;
    Ok(())
}

fn main() -> anyhow::Result<()> {
    dspy_rs::init_tracing()?;

    let rt = tokio::runtime::Runtime::new()?;
    rt.block_on(async move {
        let lm = LM::builder()
            .model("gpt-4o-mini".to_string())
            .build()
            .await?;
        Ok(())
    })
}

Inspecting history

let history = lm.inspect_history(3).await;
for entry in history {
    println!("Prompt: {}", entry.prompt);
    println!("Raw output: {:?}", entry.raw_output);
}

inspect_history requires caching to be enabled (.cache(true)); it panics on an LM built without caching. Entries are CacheEntry values served by ResponseCache; see Utils. Only tool-free calls are cached, since tool loops execute side-effectful user code.

Configuration options

All LM builder parameters have sensible defaults, so you only need to override what you need.

Parameter	Type	Default	Notes
`model`	`String`	`"openai:gpt-4o-mini"`	Supports “provider:model” format or bare model name (defaults to OpenAI)
`api_key`	`Option<String>`	`None`	Provider API key; omit for local servers
`base_url`	`Option<String>`	`None`	Custom endpoint URL; auto-detected from model provider if not provided
`temperature`	`f32`	`0.7`	Higher values increase randomness
`max_tokens`	`u32`	`512`	Upper bound on completion tokens
`max_tool_iterations`	`u32`	`10`	Upper bound on tool-loop round trips per call
`max_retries`	`u32`	`2`	Additional attempts after a transient failure (429/5xx/network/timeout); `0` disables retries
`retry_base_delay_ms`	`u64`	`250`	Base delay for exponential backoff between retries, plus up to 50% jitter
`cache`	`bool`	`false`	Enables response caching and `inspect_history` support

Example with custom settings

// API key automatically read from ANTHROPIC_API_KEY env var
let lm = LM::builder()
    .model("anthropic:claude-3-5-sonnet-20241022".to_string())
    .temperature(0.3)
    .max_tokens(1_024)
    .cache(true)
    .build()
    .await?;

Provider Support

DSRs supports multiple LLM providers through Rig. Use the provider:model format to specify which provider to use. Bare model names default to OpenAI. Supported providers:

openai - OpenAI models (requires OPENAI_API_KEY)
anthropic - Anthropic models (requires ANTHROPIC_API_KEY)
gemini - Google Gemini models (requires GEMINI_API_KEY)
groq - Groq models (requires GROQ_API_KEY)
openrouter - OpenRouter (requires OPENROUTER_API_KEY)
ollama - Local Ollama models (no API key required)

API keys are automatically read from environment variables. You only need to provide .api_key() if you want to override the default environment variable. You can also use base_url to connect to any OpenAI-compatible server (vLLM, LiteLLM, etc.).

Usage examples

// Anthropic - reads from ANTHROPIC_API_KEY env var
let lm = LM::builder()
    .model("anthropic:claude-3-5-sonnet-20241022".to_string())
    .build()
    .await?;

// Google Gemini - reads from GEMINI_API_KEY env var
let lm = LM::builder()
    .model("gemini:gemini-2.0-flash-exp".to_string())
    .build()
    .await?;

// Groq - reads from GROQ_API_KEY env var
let lm = LM::builder()
    .model("groq:mixtral-8x7b-32768".to_string())
    .build()
    .await?;

// OpenAI (or just use model name directly) - reads from OPENAI_API_KEY env var
let lm = LM::builder()
    .model("gpt-4o".to_string())  // defaults to OpenAI
    .build()
    .await?;

// Ollama (local, no API key needed)
let lm = LM::builder()
    .model("ollama:llama3".to_string())
    .build()
    .await?;

// OpenRouter - reads from OPENROUTER_API_KEY env var
let lm = LM::builder()
    .model("openrouter:anthropic/claude-3-opus".to_string())
    .build()
    .await?;

All provider integrations are powered by Rig, which handles the provider-specific API details.

Tool sets and Code Mode

LM::call accepts tools directly, but repeated calls with a fixed set of tools should build a ToolSet once and reuse it via LM::call_with_toolset. A ToolSet pre-fetches every tool definition and indexes the executors by name.

Constructor	Signature	Purpose
`ToolSet::build`	`async fn build(tools: &[Arc<dyn ToolDyn>]) -> ToolSet`	Fetches every tool definition once and indexes executors by name; duplicate names keep the first tool
`ToolSet::from_definitions`	`fn from_definitions(definitions: Vec<ToolDefinition>) -> ToolSet`	Definitions only, no executors; for caller-managed loops that execute tools themselves
`ToolSet::code_mode`	`async fn code_mode(tools: Vec<Arc<dyn ToolDyn>>, config: SandboxConfig) -> Result<ToolSet>`	Collapses the tools into a single sandboxed `run_js` tool; requires the `code-mode` feature

With ToolSet::code_mode, instead of emitting one JSON tool call per step, the model writes JavaScript against the tools as a JS API and composes their results in one execution. The returned set drops into any tool loop (LM::call_with_toolset, Predict) exactly like a normal ToolSet. It errors if two tool names mangle to the same JS identifier. See Code Mode for the full sandbox surface.

Get Started

Building Blocks

Programs as Data

Tools & Agents

Evaluation & Optimization

Utilities

LM

Responsibilities

Structure

Construction and configuration

Local server usage

Custom OpenAI-compatible endpoints

API Reference

Global vs explicit usage

Async execution and sync entry

Inspecting history

Configuration options

Example with custom settings

Provider Support

Usage examples

Tool sets and Code Mode

See also

​Responsibilities

​Structure

​Construction and configuration

​Local server usage

​Custom OpenAI-compatible endpoints

​API Reference

​Global vs explicit usage

​Async execution and sync entry

​Inspecting history

​Configuration options

​Example with custom settings

​Provider Support

​Usage examples

​Tool sets and Code Mode

​See also

Responsibilities

Structure

Construction and configuration

Local server usage

Custom OpenAI-compatible endpoints

API Reference

Global vs explicit usage

Async execution and sync entry

Inspecting history

Configuration options

Example with custom settings

Provider Support

Usage examples

Tool sets and Code Mode

See also