> ## Documentation Index
> Fetch the complete documentation index at: https://dsrs.herumbshandilya.com/llms.txt
> Use this file to discover all available pages before exploring further.

# LM

> The provider client: model selection, sampling parameters, retries, response caching, and multi-provider support

The `LM` struct is the provider client: a thin wrapper over OpenAI-compatible APIs with built-in retries, optional response caching, and history tracking. You rarely call it directly; a [predictor](/docs/components/predict) uses an [adapter](/docs/components/adapters) to format a [signature](/docs/components/signatures) and send the result through the configured LM, which keeps business logic separate from transport. Configure one globally with `configure(lm)`, or attach one per predictor with `PredictBuilder::lm(...)`.

```rust theme={null}
use dspy_rs::{init_tracing, LM};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    init_tracing()?;

    // OpenAI - API key automatically read from OPENAI_API_KEY env var
    let lm = LM::builder()
        .model("gpt-4o-mini".to_string())
        .temperature(0.7)
        .max_tokens(512)
        .build()
        .await?;

    // Or explicitly provide API key
    let lm = LM::builder()
        .model("gpt-4o-mini".to_string())
        .api_key("your-api-key".into())
        .build()
        .await?;

    Ok(())
}
```

## Responsibilities

`LM` handles three core responsibilities:

1. **Configuration** - Stores provider credentials, model selection, and inference parameters (eg: temperature)

2. **API Execution** - Takes pre-formatted `Chat` messages and executes HTTP calls to the LLM provider

3. **Response Caching** - Optionally stores input/output pairs to avoid duplicate API calls

## Structure

`LM` is built using the builder pattern. The builder collects an `LMConfig`, the serializable data half, and `build()` initializes the live client. The config holds:

* `model` - Model identifier (e.g., "gpt-4o-mini" or "openai:gpt-4o-mini")
* `api_key` - Provider API credentials (optional for local servers)
* `base_url` - API endpoint URL (optional, inferred from model provider)
* `temperature` - Sampling temperature (default: 0.7)
* `max_tokens` - Maximum completion tokens (default: 512)
* `max_tool_iterations` - Upper bound on tool-loop round trips (default: 10)
* `max_retries` - Additional attempts after a transient failure (default: 2)
* `retry_base_delay_ms` - Base delay for exponential retry backoff (default: 250)
* `cache` - Enable response caching (default: false)

The live `LM` adds:

* `client` - Internal provider client (initialized during build)
* `cache_handler` - Optional response cache (initialized during build if enabled)

Cloning an `LM` is cheap - clones share the same HTTP client and cache via `Arc`, making them ideal for concurrent use.

## Construction and configuration

The `LM::builder()` must be awaited with `.build().await` because client initialization is async.

### Local server usage

For local OpenAI-compatible servers (vLLM, Ollama, etc.), provide `base_url` without an `api_key`:

```rust theme={null}
let lm = LM::builder()
    .base_url("http://localhost:11434".to_string())
    .model("llama3".to_string())
    .build()
    .await?;
```

### Custom OpenAI-compatible endpoints

For custom endpoints requiring authentication, provide both `base_url` and `api_key`:

```rust theme={null}
let lm = LM::builder()
    .base_url("https://my-custom-api.com/v1".to_string())
    .api_key(my_api_key.into())
    .model("custom-model".to_string())
    .build()
    .await?;
```

* **Clone semantics:** `LM` implements `Clone`; clones share the underlying client and cache via `Arc`, so they see the same history while carrying their own config copy.

## API Reference

You can browse the full `LM` module reference on [docs.rs](https://docs.rs/dspy-rs/latest/dspy_rs/core/lm/index.html).

## Global vs explicit usage

* **Global:** `configure(lm)` sets the process-wide default LM used by predictors.
* **Per-instance override:** Attach an LM to a specific predictor with `PredictBuilder::lm(...)`, which bypasses the global; or build a second `LM` and call `configure(lm)` before the specific call.

## Async execution and sync entry

* **Async:** LM building and calls are `async`; prefer using an async runtime (Tokio).
* **Sync-style:** If you need a plain `fn main`, create a runtime and `block_on` the async work.

<Tabs>
  <Tab title="Async (Tokio)">
    ```rust theme={null}
    use dspy_rs::{init_tracing, LM};

    #[tokio::main]
    async fn main() -> anyhow::Result<()> {
        init_tracing()?;

        let lm = LM::builder()
            .model("gpt-4o-mini".to_string())
            .build()
            .await?;
        Ok(())
    }
    ```
  </Tab>

  <Tab title="Sync">
    ```rust theme={null}
    fn main() -> anyhow::Result<()> {
        dspy_rs::init_tracing()?;

        let rt = tokio::runtime::Runtime::new()?;
        rt.block_on(async move {
            let lm = LM::builder()
                .model("gpt-4o-mini".to_string())
                .build()
                .await?;
            Ok(())
        })
    }
    ```
  </Tab>
</Tabs>

## Inspecting history

```rust theme={null}
let history = lm.inspect_history(3).await;
for entry in history {
    println!("Prompt: {}", entry.prompt);
    println!("Raw output: {:?}", entry.raw_output);
}
```

> `inspect_history` requires caching to be enabled (`.cache(true)`); it panics on an LM built without caching. Entries are `CacheEntry` values served by `ResponseCache`; see [Utils](/docs/components/utils). Only tool-free calls are cached, since tool loops execute side-effectful user code.

## Configuration options

All `LM` builder parameters have sensible defaults, so you only need to override what you need.

| Parameter             | Type             | Default                | Notes                                                                                         |
| --------------------- | ---------------- | ---------------------- | --------------------------------------------------------------------------------------------- |
| `model`               | `String`         | `"openai:gpt-4o-mini"` | Supports "provider:model" format or bare model name (defaults to OpenAI)                      |
| `api_key`             | `Option<String>` | `None`                 | Provider API key; omit for local servers                                                      |
| `base_url`            | `Option<String>` | `None`                 | Custom endpoint URL; auto-detected from model provider if not provided                        |
| `temperature`         | `f32`            | `0.7`                  | Higher values increase randomness                                                             |
| `max_tokens`          | `u32`            | `512`                  | Upper bound on completion tokens                                                              |
| `max_tool_iterations` | `u32`            | `10`                   | Upper bound on tool-loop round trips per call                                                 |
| `max_retries`         | `u32`            | `2`                    | Additional attempts after a transient failure (429/5xx/network/timeout); `0` disables retries |
| `retry_base_delay_ms` | `u64`            | `250`                  | Base delay for exponential backoff between retries, plus up to 50% jitter                     |
| `cache`               | `bool`           | `false`                | Enables response caching and `inspect_history` support                                        |

### Example with custom settings

```rust theme={null}
// API key automatically read from ANTHROPIC_API_KEY env var
let lm = LM::builder()
    .model("anthropic:claude-3-5-sonnet-20241022".to_string())
    .temperature(0.3)
    .max_tokens(1_024)
    .cache(true)
    .build()
    .await?;
```

### Provider Support

DSRs supports multiple LLM providers through [Rig](https://github.com/0xPlaygrounds/rig). Use the `provider:model` format to specify which provider to use. Bare model names default to OpenAI.

**Supported providers:**

* `openai` - OpenAI models (requires `OPENAI_API_KEY`)
* `anthropic` - Anthropic models (requires `ANTHROPIC_API_KEY`)
* `gemini` - Google Gemini models (requires `GEMINI_API_KEY`)
* `groq` - Groq models (requires `GROQ_API_KEY`)
* `openrouter` - OpenRouter (requires `OPENROUTER_API_KEY`)
* `ollama` - Local Ollama models (no API key required)

**API keys are automatically read from environment variables.** You only need to provide `.api_key()` if you want to override the default environment variable.

You can also use `base_url` to connect to any OpenAI-compatible server (vLLM, LiteLLM, etc.).

### Usage examples

```rust theme={null}
// Anthropic - reads from ANTHROPIC_API_KEY env var
let lm = LM::builder()
    .model("anthropic:claude-3-5-sonnet-20241022".to_string())
    .build()
    .await?;

// Google Gemini - reads from GEMINI_API_KEY env var
let lm = LM::builder()
    .model("gemini:gemini-2.0-flash-exp".to_string())
    .build()
    .await?;

// Groq - reads from GROQ_API_KEY env var
let lm = LM::builder()
    .model("groq:mixtral-8x7b-32768".to_string())
    .build()
    .await?;

// OpenAI (or just use model name directly) - reads from OPENAI_API_KEY env var
let lm = LM::builder()
    .model("gpt-4o".to_string())  // defaults to OpenAI
    .build()
    .await?;

// Ollama (local, no API key needed)
let lm = LM::builder()
    .model("ollama:llama3".to_string())
    .build()
    .await?;

// OpenRouter - reads from OPENROUTER_API_KEY env var
let lm = LM::builder()
    .model("openrouter:anthropic/claude-3-opus".to_string())
    .build()
    .await?;
```

All provider integrations are powered by [Rig](https://github.com/0xPlaygrounds/rig), which handles the provider-specific API details.

## Tool sets and Code Mode

`LM::call` accepts tools directly, but repeated calls with a fixed set of tools should build a `ToolSet` once and reuse it via `LM::call_with_toolset`. A `ToolSet` pre-fetches every tool definition and indexes the executors by name.

| Constructor                 | Signature                                                                                    | Purpose                                                                                               |
| --------------------------- | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `ToolSet::build`            | `async fn build(tools: &[Arc<dyn ToolDyn>]) -> ToolSet`                                      | Fetches every tool definition once and indexes executors by name; duplicate names keep the first tool |
| `ToolSet::from_definitions` | `fn from_definitions(definitions: Vec<ToolDefinition>) -> ToolSet`                           | Definitions only, no executors; for caller-managed loops that execute tools themselves                |
| `ToolSet::code_mode`        | `async fn code_mode(tools: Vec<Arc<dyn ToolDyn>>, config: SandboxConfig) -> Result<ToolSet>` | Collapses the tools into a single sandboxed `run_js` tool; requires the `code-mode` feature           |

With `ToolSet::code_mode`, instead of emitting one JSON tool call per step, the model writes JavaScript against the tools as a JS API and composes their results in one execution. The returned set drops into any tool loop (`LM::call_with_toolset`, `Predict`) exactly like a normal `ToolSet`. It errors if two tool names mangle to the same JS identifier. See [Code Mode](/docs/components/code-mode) for the full sandbox surface.

## See also

* [Predict](/docs/components/predict)
* [Code Mode](/docs/components/code-mode)
* [Utils](/docs/components/utils)