LLMs

概览

CrewAI 通过各 provider 的原生 SDK 集成多个 LLM provider，让你可以根据具体场景灵活选择合适的模型。本指南会帮助你了解如何在 CrewAI 项目中配置并使用不同的 LLM provider。

什么是 LLM？

大语言模型（Large Language Models, LLMs）是 CrewAI agents 的核心智能。它们让 agent 能够理解上下文、做出决策，并生成类似人类的响应。你需要知道以下几点：

LLM 基础

大语言模型是基于海量文本数据训练的 AI 系统。它们为 CrewAI agents 提供智能能力，使其能够理解并生成类人文本。

上下文窗口

上下文窗口决定了一个 LLM 一次可以处理多少文本。更大的窗口（例如 128K tokens）可以容纳更多上下文，但通常也更贵、速度更慢。

Temperature

Temperature（0.0 到 1.0）控制响应的随机性。较低的值（例如 0.2）会产生更聚焦、更确定性的输出，而较高的值（例如 0.8）会提高创造性和多样性。

Provider 选择

不同的 LLM provider（例如 OpenAI、Anthropic、Google）提供不同模型，能力、定价和特性也各不相同。请选择最符合准确性、速度和成本需求的方案。

配置你的 LLM

在 CrewAI 代码中，你可以在不同位置指定要使用的模型。确定模型后，还需要为所用的每个 model provider 提供配置（例如 API key）。你的 provider 配置示例见 provider configuration examples。

1. 环境变量

最简单的入门方式。你可以通过 .env 文件或应用代码直接在环境中设置模型。如果你使用 crewai create 来初始化项目，它通常已经为你设置好了。

MODEL=model-id  # 例如 gpt-4o、gemini-2.0-flash、claude-3-sonnet-...

# 也要在这里设置你的 API keys。参见下面的 Provider
# 小节。

2. YAML 配置

创建一个 YAML 文件来定义 agent 配置。这种方式非常适合版本控制和团队协作：

researcher:
    role: Research Specialist
    goal: Conduct comprehensive research and analysis
    backstory: A dedicated research professional with years of experience
    verbose: true
    llm: provider/model-id  # 例如 openai/gpt-4o、google/gemini-2.0-flash、anthropic/claude...
    # （更多内容见下面的 provider 配置示例）

3. 直接代码

如果你需要最大灵活性，可以在 Python 代码中直接配置 LLM：

from crewai import LLM

# 基础配置
llm = LLM(model="model-id-here")  # gpt-4o、gemini-2.0-flash、anthropic/claude...

# 带详细参数的高级配置
llm = LLM(
    model="model-id-here",  # gpt-4o、gemini-2.0-flash、anthropic/claude...
    temperature=0.7,        # 更高的值会产生更有创造性的输出
    timeout=120,            # 等待响应的秒数
    max_tokens=4000,        # 响应的最大长度
    top_p=0.9,              # 核采样参数
    frequency_penalty=0.1 , # 减少重复
    presence_penalty=0.1,   # 鼓励主题多样性
    response_format={"type": "json"},  # 用于结构化输出
    seed=42                 # 用于可复现结果
)

Info

CrewAI 为 OpenAI、Anthropic、Google（Gemini API）、Azure、AWS Bedrock 和 Snowflake Cortex 提供原生 SDK 集成，无需额外安装，除了 provider 专用的可选依赖（例如 uv add "crewai[openai]"）。

其他所有 provider 都由 LiteLLM 提供支持。如果你计划使用其中任何一个，请将其作为依赖添加到项目中：

uv add 'crewai[litellm]'

Provider 配置示例

CrewAI 支持众多 LLM provider，每个 provider 都有独特的功能、认证方式和模型能力。本节提供详细示例，帮助你选择、配置并优化最适合项目需求的 LLM。

OpenAI

CrewAI 通过 OpenAI Python SDK 与 OpenAI 原生集成。

# 必需
OPENAI_API_KEY=sk-...

# 可选
OPENAI_BASE_URL=<custom-base-url>

基础用法：

from crewai import LLM

llm = LLM(
    model="openai/gpt-4o",
    api_key="your-api-key",  # 或设置 OPENAI_API_KEY
    temperature=0.7,
    max_tokens=4000
)

高级配置：

from crewai import LLM

llm = LLM(
    model="openai/gpt-4o",
    api_key="your-api-key",
    base_url="https://api.openai.com/v1",  # 可选自定义端点
    organization="org-...",  # 可选组织 ID
    project="proj_...",  # 可选项目 ID
    temperature=0.7,
    max_tokens=4000,
    max_completion_tokens=4000,  # 新模型使用
    top_p=0.9,
    frequency_penalty=0.1,
    presence_penalty=0.1,
    stop=["END"],
    seed=42,  # 用于可复现输出
    stream=True,  # 启用流式输出
    timeout=60.0,  # 请求超时（秒）
    max_retries=3,  # 最大重试次数
    logprobs=True,  # 返回 token 概率
    top_logprobs=5,  # 最可能 token 的数量
    reasoning_effort="medium"  # o1 models 使用：low, medium, high
)

结构化输出：

from pydantic import BaseModel
from crewai import LLM

class ResponseFormat(BaseModel):
    name: str
    age: int
    summary: str

llm = LLM(
    model="openai/gpt-4o",
)

支持的环境变量：

OPENAI_API_KEY：你的 OpenAI API key（必需）
OPENAI_BASE_URL：OpenAI API 的自定义 base URL（可选）

特性：

原生 function calling 支持（o1 models 除外）
基于 JSON schema 的结构化输出
支持流式输出，可实时返回响应
token 使用量跟踪
支持 stop sequences（o1 models 除外）
token 级别洞察的 log probabilities
o1 models 的 reasoning effort 控制

支持的模型：

Model	Context Window	Best For
gpt-4.1	1M tokens	Latest model with enhanced capabilities
gpt-4.1-mini	1M tokens	Efficient version with large context
gpt-4.1-nano	1M tokens	Ultra-efficient variant
gpt-4o	128,000 tokens	Optimized for speed and intelligence
gpt-4o-mini	200,000 tokens	Cost-effective with large context
gpt-4-turbo	128,000 tokens	Long-form content, document analysis
gpt-4	8,192 tokens	High-accuracy tasks, complex reasoning
o1	200,000 tokens	Advanced reasoning, complex problem-solving
o1-preview	128,000 tokens	Preview of reasoning capabilities
o1-mini	128,000 tokens	Efficient reasoning model
o3-mini	200,000 tokens	Lightweight reasoning model
o4-mini	200,000 tokens	Next-gen efficient reasoning

Responses API：

OpenAI 提供两种 API：Chat Completions（默认）和较新的 Responses API。Responses API 从设计之初就支持原生多模态 - 文本、图像、音频和函数调用都属于一等能力。它对 reasoning models 有更好的性能，并支持自动串联和内置工具等额外特性。

from crewai import LLM

# 使用 Responses API，而不是 Chat Completions
llm = LLM(
    model="openai/gpt-4o",
    api="responses",  # 启用 Responses API
    store=True,  # 为多轮对话保存响应（可选）
    auto_chain=True,  # 对 reasoning models 自动串联（可选）
)

Responses API 参数：

api：设为 "responses" 以使用 Responses API（默认："completions"）
instructions：系统级指令（仅 Responses API）
store：是否为多轮对话保存响应
previous_response_id：多轮对话中上一条 response 的 ID
include：要包含在响应中的额外数据（例如 ["reasoning.encrypted_content"]）
builtin_tools：OpenAI 内置工具列表："web_search"、"file_search"、"code_interpreter"、"computer_use"
parse_tool_outputs：返回带有已解析内置工具输出的结构化 ResponsesAPIResult
auto_chain：自动跟踪并使用 response IDs 处理多轮对话
auto_chain_reasoning：跟踪加密 reasoning 项，以满足 ZDR（Zero Data Retention）合规要求

注意： 要使用 OpenAI，请安装所需依赖：

uv add "crewai[openai]"

Meta-Llama

Meta 的 Llama API 提供对 Meta 大语言模型系列的访问。该 API 可通过 Meta Llama API 使用。

在你的 .env 文件中设置以下环境变量：

# Meta Llama API Key Configuration
LLAMA_API_KEY=LLM|your_api_key_here

在 CrewAI 项目中的使用示例：

from crewai import LLM

# 初始化 Meta Llama LLM
llm = LLM(
    model="meta_llama/Llama-4-Scout-17B-16E-Instruct-FP8",
    temperature=0.8,
    stop=["END"],
    seed=42
)

此处列出的所有模型都受支持：https://llama.developer.meta.com/docs/models/

Model ID	Input context length	Output context length	Input Modalities	Output Modalities
`meta_llama/Llama-4-Scout-17B-16E-Instruct-FP8`	128k	4028	Text, Image	Text
`meta_llama/Llama-4-Maverick-17B-128E-Instruct-FP8`	128k	4028	Text, Image	Text
`meta_llama/Llama-3.3-70B-Instruct`	128k	4028	Text	Text
`meta_llama/Llama-3.3-8B-Instruct`	128k	4028	Text	Text

注意： 此 provider 使用 LiteLLM。请将其添加为依赖：

uv add 'crewai[litellm]'

Snowflake Cortex

CrewAI 通过 Snowflake Cortex 的 OpenAI 兼容 Chat Completions endpoint 提供原生集成。这避免了对 snowflake/... models 的 LiteLLM 回退。Snowflake Cortex 目前在 CrewAI 中只支持 Chat Completions，因此请使用默认 api 模式，不要设置 api="responses"。

# Required
SNOWFLAKE_PAT=<your-programmatic-access-token>
SNOWFLAKE_ACCOUNT_URL=https://<account-identifier>.snowflakecomputing.com

# Alternative account configuration
SNOWFLAKE_ACCOUNT=<account-identifier>

Basic Usage:

from crewai import LLM

llm = LLM(
    model="snowflake/openai-gpt-4.1",
    temperature=0.7,
    max_completion_tokens=1024,
)

Claude Models on Cortex:

from crewai import LLM

llm = LLM(
    model="snowflake/claude-sonnet-4-5",
    max_completion_tokens=1024,
    stream=True,
)

Supported Environment Variables:

SNOWFLAKE_PAT, SNOWFLAKE_TOKEN, or SNOWFLAKE_JWT: token used as the Bearer credential
SNOWFLAKE_ACCOUNT_URL: full Snowflake account URL
SNOWFLAKE_ACCOUNT, SNOWFLAKE_ACCOUNT_ID, or SNOWFLAKE_ACCOUNT_IDENTIFIER: account identifier used to build the account URL

Snowflake REST requests use the user’s default Snowflake role. Make sure that role has SNOWFLAKE.CORTEX_USER or SNOWFLAKE.CORTEX_REST_API_USER. Database, schema, warehouse, and explicit role parameters are not required by the Cortex REST Chat Completions endpoint.

Features:

Native provider selection with model="snowflake/<model-name>"
Streaming and non-streaming Chat Completions only; api="responses" is not supported
Token usage tracking
Function calling for Snowflake-hosted OpenAI and Claude models
Automatic removal of invalid trailing assistant prefill for Snowflake Claude models

Anthropic

CrewAI 通过 Anthropic Python SDK 与 Anthropic 原生集成。

# Required
ANTHROPIC_API_KEY=sk-ant-...

Basic Usage:

from crewai import LLM

llm = LLM(
    model="anthropic/claude-3-5-sonnet-20241022",
    api_key="your-api-key",  # Or set ANTHROPIC_API_KEY
    max_tokens=4096  # Required for Anthropic
)

Advanced Configuration:

from crewai import LLM

llm = LLM(
    model="anthropic/claude-3-5-sonnet-20241022",
    api_key="your-api-key",
    base_url="https://api.anthropic.com",  # Optional custom endpoint
    temperature=0.7,
    max_tokens=4096,  # Required parameter
    top_p=0.9,
    stop_sequences=["END", "STOP"],  # Anthropic uses stop_sequences
    stream=True,  # Enable streaming
    timeout=60.0,  # Request timeout in seconds
    max_retries=3  # Maximum retry attempts
)

Extended Thinking (Claude Sonnet 4 and Beyond):

CrewAI 支持 Anthropic 的 Extended Thinking 功能，它允许 Claude 在响应前以更接近人类的方式思考问题。这对于复杂推理、分析和问题求解任务特别有用。

from crewai import LLM

# 启用 extended thinking，使用默认设置
llm = LLM(
    model="anthropic/claude-sonnet-4",
    thinking={"type": "enabled"},
    max_tokens=10000
)

# 使用预算控制配置 thinking
llm = LLM(
    model="anthropic/claude-sonnet-4",
    thinking={
        "type": "enabled",
        "budget_tokens": 5000  # 限制 thinking tokens
    },
    max_tokens=10000
)

Thinking 配置选项：

type：设为 "enabled" 以启用 extended thinking 模式
budget_tokens（可选）：thinking 可使用的最大 token 数（有助于控制成本）

支持 Extended Thinking 的模型：

claude-sonnet-4 及更高版本
claude-3-7-sonnet（支持 extended thinking）

何时使用 Extended Thinking：

复杂推理和多步骤问题求解
数学计算与证明
代码分析与调试
战略规划和决策制定
研究和分析任务

注意： Extended thinking 会消耗额外 token，但对复杂任务的响应质量通常会显著提升。

支持的环境变量：

ANTHROPIC_API_KEY：你的 Anthropic API key（必需）

特性：

Claude 3+ models 的原生 tool use 支持
Claude Sonnet 4+ 的 Extended Thinking 支持
实时响应流式输出
自动处理 system message
用 stop sequences 控制输出
token 使用量跟踪
多轮 tool use 对话

重要说明：

max_tokens 是所有 Anthropic models 的必需参数
Claude 使用 stop_sequences 而不是 stop
system messages 与 conversation messages 分开处理
第一条消息必须来自用户（系统会自动处理）
消息必须在 user 与 assistant 之间交替

支持的模型：

Model	Context Window	Best For
claude-sonnet-4	200,000 tokens	Latest with extended thinking capabilities
claude-3-7-sonnet	200,000 tokens	Advanced reasoning and agentic tasks
claude-3-5-sonnet-20241022	200,000 tokens	Latest Sonnet with best performance
claude-3-5-haiku	200,000 tokens	Fast, compact model for quick responses
claude-3-opus	200,000 tokens	Most capable for complex tasks
claude-3-sonnet	200,000 tokens	Balanced intelligence and speed
claude-3-haiku	200,000 tokens	Fastest for simple tasks
claude-2.1	200,000 tokens	Extended context, reduced hallucinations
claude-2	100,000 tokens	Versatile model for various tasks
claude-instant	100,000 tokens	Fast, cost-effective for everyday tasks

注意： 要使用 Anthropic，请安装所需依赖：

uv add "crewai[anthropic]"

Google (Gemini API)

CrewAI 通过 Google Gen AI Python SDK 与 Google Gemini 原生集成。

在 .env 文件中设置 API key。如果你需要 key，可以查看 AI Studio。

# Required (one of the following)
GOOGLE_API_KEY=<your-api-key>
GEMINI_API_KEY=<your-api-key>

# For Vertex AI Express mode (API key authentication)
GOOGLE_GENAI_USE_VERTEXAI=true
GOOGLE_API_KEY=<your-api-key>

# For Vertex AI with service account
GOOGLE_CLOUD_PROJECT=<your-project-id>
GOOGLE_CLOUD_LOCATION=<location>  # Defaults to us-central1

Basic Usage:

from crewai import LLM

llm = LLM(
    model="gemini/gemini-2.0-flash",
    api_key="your-api-key",  # Or set GOOGLE_API_KEY/GEMINI_API_KEY
    temperature=0.7
)

Advanced Configuration:

from crewai import LLM

llm = LLM(
    model="gemini/gemini-2.5-flash",
    api_key="your-api-key",
    temperature=0.7,
    top_p=0.9,
    top_k=40,  # Top-k sampling parameter
    max_output_tokens=8192,
    stop_sequences=["END", "STOP"],
    stream=True,  # Enable streaming
    safety_settings={
        "HARM_CATEGORY_HARASSMENT": "BLOCK_NONE",
        "HARM_CATEGORY_HATE_SPEECH": "BLOCK_NONE"
    }
)

Vertex AI Express Mode (API Key Authentication):

Vertex AI Express mode allows you to use Vertex AI with simple API key authentication instead of service account credentials. This is the quickest way to get started with Vertex AI.

To enable Express mode, set both environment variables in your .env file:

GOOGLE_GENAI_USE_VERTEXAI=true
GOOGLE_API_KEY=<your-api-key>

Then use the LLM as usual:

from crewai import LLM

llm = LLM(
    model="gemini/gemini-2.0-flash",
    temperature=0.7
)

Vertex AI Configuration (Service Account):

from crewai import LLM

llm = LLM(
    model="gemini/gemini-1.5-pro",
    project="your-gcp-project-id",
    location="us-central1"  # GCP region
)

Supported Environment Variables:

GOOGLE_API_KEY or GEMINI_API_KEY: Your Google API key (required for Gemini API and Vertex AI Express mode)
GOOGLE_GENAI_USE_VERTEXAI: Set to true to use Vertex AI (required for Express mode)
GOOGLE_CLOUD_PROJECT: Google Cloud project ID (for Vertex AI with service account)
GOOGLE_CLOUD_LOCATION: GCP location (defaults to us-central1)

Features:

Native function calling support for Gemini 1.5+ and 2.x models
Streaming support for real-time responses
Multimodal capabilities (text, images, video)
Safety settings configuration
Support for both Gemini API and Vertex AI
Automatic system instruction handling
Token usage tracking

Gemini Models:

Google offers a range of powerful models optimized for different use cases.

Model	Context Window	Best For
gemini-2.5-flash	1M tokens	Adaptive thinking, cost efficiency
gemini-2.5-pro	1M tokens	Enhanced thinking and reasoning, multimodal understanding
gemini-2.0-flash	1M tokens	Next generation features, speed, thinking
gemini-2.0-flash-thinking	32,768 tokens	Advanced reasoning with thinking process
gemini-2.0-flash-lite	1M tokens	Cost efficiency and low latency
gemini-1.5-pro	2M tokens	Best performing, logical reasoning, coding
gemini-1.5-flash	1M tokens	Balanced multimodal model, good for most tasks
gemini-1.5-flash-8b	1M tokens	Fastest, most cost-efficient
gemini-1.0-pro	32,768 tokens	Earlier generation model

Gemma Models:

The Gemini API also supports Gemma models hosted on Google infrastructure.

Model	Context Window	Best For
gemma-3-1b	32,000 tokens	Ultra-lightweight tasks
gemma-3-4b	128,000 tokens	Efficient general-purpose tasks
gemma-3-12b	128,000 tokens	Balanced performance and efficiency
gemma-3-27b	128,000 tokens	High-performance tasks

Note: To use Google Gemini, install the required dependencies:

uv add "crewai[google-genai]"

The full list of models is available in the Gemini model docs.

Google (Vertex AI)

Get credentials from your Google Cloud Console and save it to a JSON file, then load it with the following code:

import json

file_path = 'path/to/vertex_ai_service_account.json'

# Load the JSON file
with open(file_path, 'r') as file:
    vertex_credentials = json.load(file)

# Convert the credentials to a JSON string
vertex_credentials_json = json.dumps(vertex_credentials)

Example usage in your CrewAI project:

from crewai import LLM

llm = LLM(
    model="gemini-1.5-pro-latest", # or vertex_ai/gemini-1.5-pro-latest
    temperature=0.7,
    vertex_credentials=vertex_credentials_json
)

Google offers a range of powerful models optimized for different use cases:

Model	Context Window	Best For
gemini-2.5-flash-preview-04-17	1M tokens	Adaptive thinking, cost efficiency
gemini-2.5-pro-preview-05-06	1M tokens	Enhanced thinking and reasoning, multimodal understanding, advanced coding, and more
gemini-2.0-flash	1M tokens	Next generation features, speed, thinking, and realtime streaming
gemini-2.0-flash-lite	1M tokens	Cost efficiency and low latency
gemini-1.5-flash	1M tokens	Balanced multimodal model, good for most tasks
gemini-1.5-flash-8B	1M tokens	Fastest, most cost-efficient, good for high-frequency tasks
gemini-1.5-pro	2M tokens	Best performing, wide variety of reasoning tasks including logical reasoning, coding, and creative collaboration

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Azure

CrewAI provides native integration with Azure AI Inference and Azure OpenAI through the Azure AI Inference Python SDK.

# Required
AZURE_API_KEY=<your-api-key>
AZURE_ENDPOINT=<your-endpoint-url>

# Optional
AZURE_API_VERSION=<api-version>  # Defaults to 2024-06-01

Endpoint URL Formats:

For Azure OpenAI deployments:

https://<resource-name>.openai.azure.com/openai/deployments/<deployment-name>

For Azure AI Inference endpoints:

https://<resource-name>.inference.azure.com

Basic Usage:

llm = LLM(
    model="azure/gpt-4",
    api_key="<your-api-key>",  # Or set AZURE_API_KEY
    endpoint="<your-endpoint-url>",
    api_version="2024-06-01"
)

Advanced Configuration:

llm = LLM(
    model="azure/gpt-4o",
    temperature=0.7,
    max_tokens=4000,
    top_p=0.9,
    frequency_penalty=0.0,
    presence_penalty=0.0,
    stop=["END"],
    stream=True,
    timeout=60.0,
    max_retries=3
)

Supported Environment Variables:

AZURE_API_KEY: Your Azure API key (required)
AZURE_ENDPOINT: Your Azure endpoint URL (required, also checks AZURE_OPENAI_ENDPOINT and AZURE_API_BASE)
AZURE_API_VERSION: API version (optional, defaults to 2024-06-01)

Features:

Native function calling support for Azure OpenAI models (gpt-4, gpt-4o, gpt-3.5-turbo, etc.)
Streaming support for real-time responses
Automatic endpoint URL validation and correction
Comprehensive error handling with retry logic
Token usage tracking

Note: To use Azure AI Inference, install the required dependencies:

uv add "crewai[azure-ai-inference]"

AWS Bedrock

CrewAI provides native integration with AWS Bedrock through the boto3 SDK using the Converse API.

# Required
AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-key>

# Optional
AWS_SESSION_TOKEN=<your-session-token>  # For temporary credentials
AWS_DEFAULT_REGION=<your-region>  # Defaults to us-east-1
AWS_REGION_NAME=<your-region>  # Alternative configuration for backwards compatibility with LiteLLM. Defaults to us-east-1

Basic Usage:

from crewai import LLM

llm = LLM(
    model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
    region_name="us-east-1"
)

Advanced Configuration:

from crewai import LLM

llm = LLM(
    model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
    aws_access_key_id="your-access-key",  # Or set AWS_ACCESS_KEY_ID
    aws_secret_access_key="your-secret-key",  # Or set AWS_SECRET_ACCESS_KEY
    aws_session_token="your-session-token",  # For temporary credentials
    region_name="us-east-1",
    temperature=0.7,
    max_tokens=4096,
    top_p=0.9,
    top_k=250,  # For Claude models
    stop_sequences=["END", "STOP"],
    stream=True,  # Enable streaming
    guardrail_config={  # Optional content filtering
        "guardrailIdentifier": "your-guardrail-id",
        "guardrailVersion": "1"
    },
    additional_model_request_fields={  # Model-specific parameters
        "top_k": 250
    }
)

Supported Environment Variables:

AWS_ACCESS_KEY_ID: AWS access key (required)
AWS_SECRET_ACCESS_KEY: AWS secret key (required)
AWS_SESSION_TOKEN: AWS session token for temporary credentials (optional)
AWS_DEFAULT_REGION: AWS region (defaults to us-east-1)
AWS_REGION_NAME: AWS region (defaults to us-east-1). Alternative configuration for backwards compatibility with LiteLLM

Features:

Native tool calling support via Converse API
Streaming and non-streaming responses
Comprehensive error handling with retry logic
Guardrail configuration for content filtering
Model-specific parameters via additional_model_request_fields
Token usage tracking and stop reason logging
Support for all Bedrock foundation models
Automatic conversation format handling

Important Notes:

Uses the modern Converse API for unified model access
Automatic handling of model-specific conversation requirements
System messages are handled separately from conversation
First message must be from user (automatically handled)
Some models (like Cohere) require conversation to end with user message

Amazon Bedrock is a managed service that provides access to multiple foundation models from top AI companies through a unified API.

Model	Context Window	Best For
Amazon Nova Pro	Up to 300k tokens	High-performance, model balancing accuracy, speed, and cost-effectiveness across diverse tasks.
Amazon Nova Micro	Up to 128k tokens	High-performance, cost-effective text-only model optimized for lowest latency responses.
Amazon Nova Lite	Up to 300k tokens	High-performance, affordable multimodal processing for images, video, and text with real-time capabilities.
Claude 3.7 Sonnet	Up to 128k tokens	High-performance, best for complex reasoning, coding & AI agents
Claude 3.5 Sonnet v2	Up to 200k tokens	State-of-the-art model specialized in software engineering, agentic capabilities, and computer interaction at optimized cost.
Claude 3.5 Sonnet	Up to 200k tokens	High-performance model delivering superior intelligence and reasoning across diverse tasks with optimal speed-cost balance.
Claude 3.5 Haiku	Up to 200k tokens	Fast, compact multimodal model optimized for quick responses and seamless human-like interactions
Claude 3 Sonnet	Up to 200k tokens	Multimodal model balancing intelligence and speed for high-volume deployments.
Claude 3 Haiku	Up to 200k tokens	Compact, high-speed multimodal model optimized for quick responses and natural conversational interactions
Claude 3 Opus	Up to 200k tokens	Most advanced multimodal model exceling at complex tasks with human-like reasoning and superior contextual understanding.
Claude 2.1	Up to 200k tokens	Enhanced version with expanded context window, improved reliability, and reduced hallucinations for long-form and RAG applications
Claude	Up to 100k tokens	Versatile model excelling in sophisticated dialogue, creative content, and precise instruction following.
Claude Instant	Up to 100k tokens	Fast, cost-effective model for everyday tasks like dialogue, analysis, summarization, and document Q&A
Llama 3.1 405B Instruct	Up to 128k tokens	Advanced LLM for synthetic data generation, distillation, and inference for chatbots, coding, and domain-specific tasks.
Llama 3.1 70B Instruct	Up to 128k tokens	Powers complex conversations with superior contextual understanding, reasoning and text generation.
Llama 3.1 8B Instruct	Up to 128k tokens	Advanced state-of-the-art model with language understanding, superior reasoning, and text generation.
Llama 3 70B Instruct	Up to 8k tokens	Powers complex conversations with superior contextual understanding, reasoning and text generation.
Llama 3 8B Instruct	Up to 8k tokens	Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation.
Titan Text G1 - Lite	Up to 4k tokens	Lightweight, cost-effective model optimized for English tasks and fine-tuning with focus on summarization and content generation.
Titan Text G1 - Express	Up to 8k tokens	Versatile model for general language tasks, chat, and RAG applications with support for English and 100+ languages.
Cohere Command	Up to 4k tokens	Model specialized in following user commands and delivering practical enterprise solutions.
Jurassic-2 Mid	Up to 8,191 tokens	Cost-effective model balancing quality and affordability for diverse language tasks like Q&A, summarization, and content generation.
Jurassic-2 Ultra	Up to 8,191 tokens	Model for advanced text generation and comprehension, excelling in complex tasks like analysis and content creation.
Jamba-Instruct	Up to 256k tokens	Model with extended context window optimized for cost-effective text generation, summarization, and Q&A.
Mistral 7B Instruct	Up to 32k tokens	This LLM follows instructions, completes requests, and generates creative text.
Mistral 8x7B Instruct	Up to 32k tokens	An MOE LLM that follows instructions, completes requests, and generates creative text.
DeepSeek R1	32,768 tokens	Advanced reasoning model

Note: To use AWS Bedrock, install the required dependencies:

uv add "crewai[bedrock]"

Amazon SageMaker

AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-key>
AWS_DEFAULT_REGION=<your-region>

Example usage in your CrewAI project:

llm = LLM(
    model="sagemaker/<my-endpoint>"
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Mistral

Set the following environment variables in your .env file:

MISTRAL_API_KEY=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="mistral/mistral-large-latest",
    temperature=0.7
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Nvidia NIM

Set the following environment variables in your .env file:

NVIDIA_API_KEY=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="nvidia_nim/meta/llama3-70b-instruct",
    temperature=0.7
)

Nvidia NIM provides a comprehensive suite of models for various use cases, from general-purpose tasks to specialized applications.

Model	Context Window	Best For
nvidia/mistral-nemo-minitron-8b-8k-instruct	8,192 tokens	State-of-the-art small language model delivering superior accuracy for chatbot, virtual assistants, and content generation.
nvidia/nemotron-4-mini-hindi-4b-instruct	4,096 tokens	A bilingual Hindi-English SLM for on-device inference, tailored specifically for Hindi Language.
nvidia/llama-3.1-nemotron-70b-instruct	128k tokens	Customized for enhanced helpfulness in responses
nvidia/llama3-chatqa-1.5-8b	128k tokens	Advanced LLM to generate high-quality, context-aware responses for chatbots and search engines.
nvidia/llama3-chatqa-1.5-70b	128k tokens	Advanced LLM to generate high-quality, context-aware responses for chatbots and search engines.
nvidia/vila	128k tokens	Multi-modal vision-language model that understands text/img/video and creates informative responses
nvidia/neva-22	4,096 tokens	Multi-modal vision-language model that understands text/images and generates informative responses
nvidia/nemotron-mini-4b-instruct	8,192 tokens	General-purpose tasks
nvidia/usdcode-llama3-70b-instruct	128k tokens	State-of-the-art LLM that answers OpenUSD knowledge queries and generates USD-Python code.
nvidia/nemotron-4-340b-instruct	4,096 tokens	Creates diverse synthetic data that mimics the characteristics of real-world data.
meta/codellama-70b	100k tokens	LLM capable of generating code from natural language and vice versa.
meta/llama2-70b	4,096 tokens	Cutting-edge large language AI model capable of generating text and code in response to prompts.
meta/llama3-8b-instruct	8,192 tokens	Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation.
meta/llama3-70b-instruct	8,192 tokens	Powers complex conversations with superior contextual understanding, reasoning and text generation.
meta/llama-3.1-8b-instruct	128k tokens	Advanced state-of-the-art model with language understanding, superior reasoning, and text generation.
meta/llama-3.1-70b-instruct	128k tokens	Powers complex conversations with superior contextual understanding, reasoning and text generation.
meta/llama-3.1-405b-instruct	128k tokens	Advanced LLM for synthetic data generation, distillation, and inference for chatbots, coding, and domain-specific tasks.
meta/llama-3.2-1b-instruct	128k tokens	Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation.
meta/llama-3.2-3b-instruct	128k tokens	Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation.
meta/llama-3.2-11b-vision-instruct	128k tokens	Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation.
meta/llama-3.2-90b-vision-instruct	128k tokens	Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation.
google/gemma-7b	8,192 tokens	Cutting-edge text generation model text understanding, transformation, and code generation.
google/gemma-2b	8,192 tokens	Cutting-edge text generation model text understanding, transformation, and code generation.
google/codegemma-7b	8,192 tokens	Cutting-edge model built on Google’s Gemma-7B specialized for code generation and code completion.
google/codegemma-1.1-7b	8,192 tokens	Advanced programming model for code generation, completion, reasoning, and instruction following.
google/recurrentgemma-2b	8,192 tokens	Novel recurrent architecture based language model for faster inference when generating long sequences.
google/gemma-2-9b-it	8,192 tokens	Cutting-edge text generation model text understanding, transformation, and code generation.
google/gemma-2-27b-it	8,192 tokens	Cutting-edge text generation model text understanding, transformation, and code generation.
google/gemma-2-2b-it	8,192 tokens	Cutting-edge text generation model text understanding, transformation, and code generation.
google/deplot	512 tokens	One-shot visual language understanding model that translates images of plots into tables.
google/paligemma	8,192 tokens	Vision language model adept at comprehending text and visual inputs to produce informative responses.
mistralai/mistral-7b-instruct-v0.2	32k tokens	This LLM follows instructions, completes requests, and generates creative text.
mistralai/mixtral-8x7b-instruct-v0.1	8,192 tokens	An MOE LLM that follows instructions, completes requests, and generates creative text.
mistralai/mistral-large	4,096 tokens	Creates diverse synthetic data that mimics the characteristics of real-world data.
mistralai/mixtral-8x22b-instruct-v0.1	8,192 tokens	Creates diverse synthetic data that mimics the characteristics of real-world data.
mistralai/mistral-7b-instruct-v0.3	32k tokens	This LLM follows instructions, completes requests, and generates creative text.
nv-mistralai/mistral-nemo-12b-instruct	128k tokens	Most advanced language model for reasoning, code, multilingual tasks; runs on a single GPU.
mistralai/mamba-codestral-7b-v0.1	256k tokens	Model for writing and interacting with code across a wide range of programming languages and tasks.
microsoft/phi-3-mini-128k-instruct	128K tokens	Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills.
microsoft/phi-3-mini-4k-instruct	4,096 tokens	Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills.
microsoft/phi-3-small-8k-instruct	8,192 tokens	Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills.
microsoft/phi-3-small-128k-instruct	128K tokens	Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills.
microsoft/phi-3-medium-4k-instruct	4,096 tokens	Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills.
microsoft/phi-3-medium-128k-instruct	128K tokens	Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills.
microsoft/phi-3.5-mini-instruct	128K tokens	Lightweight multilingual LLM powering AI applications in latency bound, memory/compute constrained environments
microsoft/phi-3.5-moe-instruct	128K tokens	Advanced LLM based on Mixture of Experts architecture to deliver compute efficient content generation
microsoft/kosmos-2	1,024 tokens	Groundbreaking multimodal model designed to understand and reason about visual elements in images.
microsoft/phi-3-vision-128k-instruct	128k tokens	Cutting-edge open multimodal model exceling in high-quality reasoning from images.
microsoft/phi-3.5-vision-instruct	128k tokens	Cutting-edge open multimodal model exceling in high-quality reasoning from images.
databricks/dbrx-instruct	12k tokens	A general-purpose LLM with state-of-the-art performance in language understanding, coding, and RAG.
snowflake/arctic	1,024 tokens	Delivers high efficiency inference for enterprise applications focused on SQL generation and coding.
aisingapore/sea-lion-7b-instruct	4,096 tokens	LLM to represent and serve the linguistic and cultural diversity of Southeast Asia
ibm/granite-8b-code-instruct	4,096 tokens	Software programming LLM for code generation, completion, explanation, and multi-turn conversion.
ibm/granite-34b-code-instruct	8,192 tokens	Software programming LLM for code generation, completion, explanation, and multi-turn conversion.
ibm/granite-3.0-8b-instruct	4,096 tokens	Advanced Small Language Model supporting RAG, summarization, classification, code, and agentic AI
ibm/granite-3.0-3b-a800m-instruct	4,096 tokens	Highly efficient Mixture of Experts model for RAG, summarization, entity extraction, and classification
mediatek/breeze-7b-instruct	4,096 tokens	Creates diverse synthetic data that mimics the characteristics of real-world data.
upstage/solar-10.7b-instruct	4,096 tokens	Excels in NLP tasks, particularly in instruction-following, reasoning, and mathematics.
writer/palmyra-med-70b-32k	32k tokens	Leading LLM for accurate, contextually relevant responses in the medical domain.
writer/palmyra-med-70b	32k tokens	Leading LLM for accurate, contextually relevant responses in the medical domain.
writer/palmyra-fin-70b-32k	32k tokens	Specialized LLM for financial analysis, reporting, and data processing
01-ai/yi-large	32k tokens	Powerful model trained on English and Chinese for diverse tasks including chatbot and creative writing.
deepseek-ai/deepseek-coder-6.7b-instruct	2k tokens	Powerful coding model offering advanced capabilities in code generation, completion, and infilling
rakuten/rakutenai-7b-instruct	1,024 tokens	Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation.
rakuten/rakutenai-7b-chat	1,024 tokens	Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation.
baichuan-inc/baichuan2-13b-chat	4,096 tokens	Support Chinese and English chat, coding, math, instruction following, solving quizzes

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

NVIDIA Nemotron

NVIDIA Nemotron models are designed for demanding agentic workloads, including complex reasoning, long-context analysis, tool use, multilingual tasks, and high-stakes RAG.

The NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 model is a frontier-scale open-weight model from NVIDIA with 550B total parameters and 55B active parameters. It uses a LatentMoE architecture that combines Mamba-2, MoE, Attention, and Multi-Token Prediction (MTP), and supports context lengths up to 1M tokens.

Hosted NVIDIA NIM usage:

NVIDIA_API_KEY=<your-api-key>

from crewai import LLM

llm = LLM(
    model="nvidia_nim/nvidia/nvidia-nemotron-3-ultra-550b-a55b",
    temperature=0.2,
    max_tokens=4096,
)

Self-hosted OpenAI-compatible endpoint:

from crewai import LLM

llm = LLM(
    model="openai/nvidia-nemotron-3-ultra-550b-a55b-nvfp4",
    base_url="https://your-nemotron-endpoint.example.com/v1",
    api_key="your-api-key",
    temperature=0.2,
    max_tokens=4096,
)

Model details:

Model	Context Window	Best For
`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4`	Up to 1M tokens	Frontier reasoning, complex agentic workflows, long-context analysis, tool use, multilingual reasoning, and high-stakes RAG

Supported languages: English, French, Spanish, Italian, German, Japanese, Korean, Hindi, Brazilian Portuguese, and Chinese.

Reasoning mode: Nemotron 3 Ultra supports configurable reasoning via its chat template using enable_thinking=True or enable_thinking=False. If you are using a hosted endpoint, check your provider’s documentation for how that flag is exposed.

For model details, license, and deployment guidance, see the NVIDIA Nemotron 3 Ultra model card.

Note: Hosted NVIDIA NIM usage uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Local NVIDIA NIM Deployed using WSL2

NVIDIA NIM enables you to run powerful LLMs locally on your Windows machine using WSL2 (Windows Subsystem for Linux). This approach allows you to leverage your NVIDIA GPU for private, secure, and cost-effective AI inference without relying on cloud services. Perfect for development, testing, or production scenarios where data privacy or offline capabilities are required.

Here is a step-by-step guide to setting up a local NVIDIA NIM model:

Follow installation instructions from NVIDIA Website
Install the local model. For Llama 3.1-8b follow instructions
Configure your crewai local models:

from crewai.llm import LLM

local_nvidia_nim_llm = LLM(
    model="openai/meta/llama-3.1-8b-instruct", # it's an openai-api compatible model
    base_url="http://localhost:8000/v1",
    api_key="<your_api_key|any text if you have not configured it>", # api_key is required, but you can use any text
)

# Then you can use it in your crew:

@CrewBase
class MyCrew():
    # ...

    @agent
    def researcher(self) -> Agent:
        return Agent(
            config=self.agents_config['researcher'], # type: ignore[index]
            llm=local_nvidia_nim_llm
        )

    # ...

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Groq

Set the following environment variables in your .env file:

GROQ_API_KEY=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="groq/llama-3.2-90b-text-preview",
    temperature=0.7
)

Model	Context Window	Best For
Llama 3.1 70B/8B	131,072 tokens	High-performance, large context tasks
Llama 3.2 Series	8,192 tokens	General-purpose tasks
Mixtral 8x7B	32,768 tokens	Balanced performance and context

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

IBM watsonx.ai

Set the following environment variables in your .env file:

# Required
WATSONX_URL=<your-url>
WATSONX_APIKEY=<your-apikey>
WATSONX_PROJECT_ID=<your-project-id>

# Optional
WATSONX_TOKEN=<your-token>
WATSONX_DEPLOYMENT_SPACE_ID=<your-space-id>

Example usage in your CrewAI project:

llm = LLM(
    model="watsonx/meta-llama/llama-3-1-70b-instruct",
    base_url="https://api.watsonx.ai/v1"
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Ollama (Local LLMs)

Install Ollama: ollama.ai
Run a model: ollama run llama3
Configure:

llm = LLM(
    model="ollama/llama3:70b",
    base_url="http://localhost:11434"
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Fireworks AI

Set the following environment variables in your .env file:

FIREWORKS_API_KEY=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="fireworks_ai/accounts/fireworks/models/llama-v3-70b-instruct",
    temperature=0.7
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Perplexity AI

Set the following environment variables in your .env file:

PERPLEXITY_API_KEY=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="llama-3.1-sonar-large-128k-online",
    base_url="https://api.perplexity.ai/"
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Hugging Face

Set the following environment variables in your .env file:

HF_TOKEN=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct"
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

SambaNova

Set the following environment variables in your .env file:

SAMBANOVA_API_KEY=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="sambanova/Meta-Llama-3.1-8B-Instruct",
    temperature=0.7
)

Model	Context Window	Best For
Llama 3.1 70B/8B	Up to 131,072 tokens	High-performance, large context tasks
Llama 3.1 405B	8,192 tokens	High-performance and output quality
Llama 3.2 Series	8,192 tokens	General-purpose, multimodal tasks
Llama 3.3 70B	Up to 131,072 tokens	High-performance and output quality
Qwen2 familly	8,192 tokens	High-performance and output quality

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Cerebras

Set the following environment variables in your .env file:

# Required
CEREBRAS_API_KEY=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="cerebras/llama3.1-70b",
    temperature=0.7,
    max_tokens=8192
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Open Router

Set the following environment variables in your .env file:

OPENROUTER_API_KEY=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="openrouter/deepseek/deepseek-r1",
    base_url="https://openrouter.ai/api/v1",
    api_key=OPENROUTER_API_KEY
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Nebius AI Studio

Set the following environment variables in your .env file:

NEBIUS_API_KEY=<your-api-key>

Example usage in your CrewAI project:

llm = LLM(
    model="nebius/Qwen/Qwen3-30B-A3B"
)

Note: This provider uses LiteLLM. Add it as a dependency to your project:

uv add 'crewai[litellm]'

Streaming Responses

CrewAI supports streaming responses from LLMs, allowing your application to receive and process outputs in real-time as they’re generated.

Basic Setup

Enable streaming by setting the stream parameter to True when initializing your LLM:

from crewai import LLM

# Create an LLM with streaming enabled
llm = LLM(
    model="openai/gpt-4o",
    stream=True  # Enable streaming
)

When streaming is enabled, responses are delivered in chunks as they’re generated, creating a more responsive user experience.

Event Handling

CrewAI emits events for each chunk received during streaming:

from crewai.events import (
  LLMStreamChunkEvent
)
from crewai.events import BaseEventListener

class MyCustomListener(BaseEventListener):
    def setup_listeners(self, crewai_event_bus):
        @crewai_event_bus.on(LLMStreamChunkEvent)
        def on_llm_stream_chunk(self, event: LLMStreamChunkEvent):
          # Process each chunk as it arrives
          print(f"Received chunk: {event.chunk}")

my_listener = MyCustomListener()

Agent & Task Tracking

All LLM events in CrewAI include agent and task information, allowing you to track and filter LLM interactions by specific agents or tasks:

from crewai import LLM, Agent, Task, Crew
from crewai.events import LLMStreamChunkEvent
from crewai.events import BaseEventListener

class MyCustomListener(BaseEventListener):
    def setup_listeners(self, crewai_event_bus):
        @crewai_event_bus.on(LLMStreamChunkEvent)
        def on_llm_stream_chunk(source, event):
            if researcher.id == event.agent_id:
                print("\n==============\n Got event:", event, "\n==============\n")


my_listener = MyCustomListener()

llm = LLM(model="gpt-4o-mini", temperature=0, stream=True)

researcher = Agent(
    role="About User",
    goal="You know everything about the user.",
    backstory="""You are a master at understanding people and their preferences.""",
    llm=llm,
)

search = Task(
    description="Answer the following questions about the user: {question}",
    expected_output="An answer to the question.",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[search])

result = crew.kickoff(
    inputs={"question": "..."}
)

Async LLM Calls

CrewAI supports asynchronous LLM calls for improved performance and concurrency in your AI workflows. Async calls allow you to run multiple LLM requests concurrently without blocking, making them ideal for high-throughput applications and parallel agent operations.

Basic Usage

Use the acall method for asynchronous LLM requests:

import asyncio
from crewai import LLM

async def main():
    llm = LLM(model="openai/gpt-4o")

    # Single async call
    response = await llm.acall("What is the capital of France?")
    print(response)

asyncio.run(main())

The acall method supports all the same parameters as the synchronous call method, including messages, tools, and callbacks.

With Streaming

Combine async calls with streaming for real-time concurrent responses:

import asyncio
from crewai import LLM

async def stream_async():
    llm = LLM(model="openai/gpt-4o", stream=True)

    response = await llm.acall("Write a short story about AI")

    print(response)

asyncio.run(stream_async())

Structured LLM Calls

CrewAI supports structured responses from LLM calls by allowing you to define a response_format using a Pydantic model. This enables the framework to automatically parse and validate the output, making it easier to integrate the response into your application without manual post-processing.

For example, you can define a Pydantic model to represent the expected response structure and pass it as the response_format when instantiating the LLM. The model will then be used to convert the LLM output into a structured Python object.

from crewai import LLM

class Dog(BaseModel):
    name: str
    age: int
    breed: str


llm = LLM(model="gpt-4o", response_format=Dog)

response = llm.call(
    "Analyze the following messages and return the name, age, and breed. "
    "Meet Kona! She is 3 years old and is a black german shepherd."
)
print(response)

# Output:
# Dog(name='Kona', age=3, breed='black german shepherd')

Advanced Features and Optimization

Learn how to get the most out of your LLM configuration:

Context Window Management

CrewAI includes smart context management features:

from crewai import LLM

# CrewAI automatically handles:
# 1. Token counting and tracking
# 2. Content summarization when needed
# 3. Task splitting for large contexts

llm = LLM(
    model="gpt-4",
    max_tokens=4000,  # Limit response length
)

Performance Optimization

Token Usage Optimization

Choose the right context window for your task:
- Small tasks (up to 4K tokens): Standard models
- Medium tasks (between 4K-32K): Enhanced models
- Large tasks (over 32K): Large context models
```
# Configure model with appropriate settings
llm = LLM(
    model="openai/gpt-4-turbo-preview",
    temperature=0.7,    # Adjust based on task
    max_tokens=4096,    # Set based on output needs
    timeout=300        # Longer timeout for complex tasks
)
```
Tip
- Lower temperature (0.1 to 0.3) for factual responses
- Higher temperature (0.7 to 0.9) for creative tasks
Best Practices
1. Monitor token usage
2. Implement rate limiting
3. Use caching when possible
4. Set appropriate max_tokens limits

Drop Additional Parameters

CrewAI internally uses native sdks for LLM calls, which allows you to drop additional parameters that are not needed for your specific use case. This can help simplify your code and reduce the complexity of your LLM configuration. For example, if you don’t need to send the stop parameter, you can simply omit it from your LLM call:

from crewai import LLM
import os

os.environ["OPENAI_API_KEY"] = "<api-key>"

o3_llm = LLM(
    model="o3",
    drop_params=True,
    additional_drop_params=["stop"]
)

Transport Interceptors

CrewAI provides message interceptors for several providers, allowing you to hook into request/response cycles at the transport layer.

Supported Providers:

✅ OpenAI
✅ Anthropic

Basic Usage:

import httpx
from crewai import LLM
from crewai.llms.hooks import BaseInterceptor

class CustomInterceptor(BaseInterceptor[httpx.Request, httpx.Response]):
"""Custom interceptor to modify requests and responses."""

def on_outbound(self, request: httpx.Request) -> httpx.Request:
    """Print request before sending to the LLM provider."""
    print(request)
    return request

def on_inbound(self, response: httpx.Response) -> httpx.Response:
    """Process response after receiving from the LLM provider."""
    print(f"Status: {response.status_code}")
    print(f"Response time: {response.elapsed}")
    return response

# Use the interceptor with an LLM
llm = LLM(
model="openai/gpt-4o",
interceptor=CustomInterceptor()
)

Important Notes:

Both methods must return the received object or type of object.
Modifying received objects may result in unexpected behavior or application crashes.
Not all providers support interceptors - check the supported providers list above

Common Issues and Solutions

Authentication

# OpenAI
OPENAI_API_KEY=sk-...

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...

Model Names

# Correct
llm = LLM(model="openai/gpt-4")

# Incorrect
llm = LLM(model="gpt-4")

Context Length

# Large context model
llm = LLM(model="openai/gpt-4o")  # 128K tokens