Connect to any LLM

Connect CrewAI to LLMs

CrewAI connects to LLMs through native SDK integrations for the most popular providers (OpenAI, Anthropic, Google Gemini, Azure, and AWS Bedrock), and uses LiteLLM as a flexible fallback for all other providers.

Note

By default, CrewAI uses the gpt-4o-mini model. This is determined by the OPENAI_MODEL_NAME environment variable, which defaults to “gpt-4o-mini” if not set. You can easily configure your agents to use a different model or provider as described in this guide.

Supported Providers

LiteLLM supports a wide range of providers, including but not limited to:

• OpenAI
• Anthropic
• Google (Vertex AI, Gemini)
• Azure OpenAI
• AWS (Bedrock, SageMaker)
• Cohere
• VoyageAI
• Hugging Face
• Ollama
• Mistral AI
• Replicate
• Together AI
• AI21
• Cloudflare Workers AI
• DeepInfra
• Groq
• SambaNova
• Nebius AI Studio
• NVIDIA NIMs
• And many more!

For a complete and up-to-date list of supported providers, please refer to the LiteLLM Providers documentation.

Info

To use any provider not covered by a native integration, add LiteLLM as a dependency to your project:

uv add 'crewai[litellm]'

Native providers (OpenAI, Anthropic, Google Gemini, Azure, AWS Bedrock) use their own SDK extras — see the Provider Configuration Examples.

Changing the LLM

To use a different LLM with your CrewAI agents, you have several options:

Using a String Identifier

Pass the model name as a string when initializing the agent:

Code

from crewai import Agent

# Using OpenAI's GPT-4
openai_agent = Agent(
    role='OpenAI Expert',
    goal='Provide insights using GPT-4',
    backstory="An AI assistant powered by OpenAI's latest model.",
    llm='gpt-4'
)

# Using Anthropic's Claude
claude_agent = Agent(
    role='Anthropic Expert',
    goal='Analyze data using Claude',
    backstory="An AI assistant leveraging Anthropic's language model.",
    llm='claude-2'
)

Using the LLM Class

For more detailed configuration, use the LLM class:

Code

from crewai import Agent, LLM

llm = LLM(
    model="gpt-4",
    temperature=0.7,
    base_url="https://api.openai.com/v1",
    api_key="your-api-key-here"
)

agent = Agent(
    role='Customized LLM Expert',
    goal='Provide tailored responses',
    backstory="An AI assistant with custom LLM settings.",
    llm=llm
)

Configuration Options

When configuring an LLM for your agent, you have access to a wide range of parameters:

Parameter	Type	Description
model	str	The name of the model to use (e.g., “gpt-4”, “claude-2”)
temperature	float	Controls randomness in output (0.0 to 1.0)
max_tokens	int	Maximum number of tokens to generate
top_p	float	Controls diversity of output (0.0 to 1.0)
frequency_penalty	float	Penalizes new tokens based on their frequency in the text so far
presence_penalty	float	Penalizes new tokens based on their presence in the text so far
stop	str, List[str]	Sequence(s) to stop generation
base_url	str	The base URL for the API endpoint
api_key	str	Your API key for authentication

For a complete list of parameters and their descriptions, refer to the LLM class documentation.

Connecting to OpenAI-Compatible LLMs

You can connect to OpenAI-compatible LLMs using either environment variables or by setting specific attributes on the LLM class:

Using Environment Variables

Code

import os

os.environ["OPENAI_API_KEY"] = "your-api-key"
os.environ["OPENAI_API_BASE"] = "https://api.your-provider.com/v1"
os.environ["OPENAI_MODEL_NAME"] = "your-model-name"

import os

# Example using Gemini's OpenAI-compatible API.
os.environ["OPENAI_API_KEY"] = "your-gemini-key"  # Should start with AIza...
os.environ["OPENAI_API_BASE"] = "https://generativelanguage.googleapis.com/v1beta/openai/"
os.environ["OPENAI_MODEL_NAME"] = "openai/gemini-2.0-flash"  # Add your Gemini model here, under openai/

Using LLM Class Attributes

Code

llm = LLM(
    model="custom-model-name",
    api_key="your-api-key",
    base_url="https://api.your-provider.com/v1"
)
agent = Agent(llm=llm, ...)

# Example using Gemini's OpenAI-compatible API
llm = LLM(
    model="openai/gemini-2.0-flash",
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
    api_key="your-gemini-key",  # Should start with AIza...
)
agent = Agent(llm=llm, ...)

Using Local Models with Ollama

For local models like those provided by Ollama:

1. Download and install Ollama

   Click here to download and install Ollama

2. Pull the desired model

   For example, run ollama pull llama3.2 to download the model.

3. Configure your agent
   Code

       agent = Agent(
           role='Local AI Expert',
           goal='Process information using a local model',
           backstory="An AI assistant running on local hardware.",
           llm=LLM(model="ollama/llama3.2", base_url="http://localhost:11434")
       )

Changing the Base API URL

You can change the base API URL for any LLM provider by setting the base_url parameter:

llm = LLM(
    model="custom-model-name",
    base_url="https://api.your-provider.com/v1",
    api_key="your-api-key"
)
agent = Agent(llm=llm, ...)

This is particularly useful when working with OpenAI-compatible APIs or when you need to specify a different endpoint for your chosen provider.

Conclusion

By leveraging LiteLLM, CrewAI offers seamless integration with a vast array of LLMs. This flexibility allows you to choose the most suitable model for your specific needs, whether you prioritize performance, cost-efficiency, or local deployment. Remember to consult the LiteLLM documentation for the most up-to-date information on supported models and configuration options.