Skip to main content

LiteLLM Multi-Provider Integration

LiteLLM steps provide a unified interface to 100+ AI providers including OpenAI, Anthropic, Google, Azure, and local models. This integration offers maximum flexibility and vendor independence for AI workflows.

Available Steps (6)​

litellm_chat​

Chat completions with any supported AI provider using a consistent interface.

Activity Name: litellm_chat

Use Cases: Conversational AI, text generation, content creation, Q&A systems

litellm_vision​

Image analysis using vision-capable models from multiple providers.

Activity Name: litellm_vision

Use Cases: Image analysis, visual content description, accessibility alt-text generation

litellm_embeddings​

Generate text embeddings using various embedding models and providers.

Activity Name: litellm_embeddings

Use Cases: Semantic search, similarity analysis, vector database preparation

litellm_image_generation​

Create images using DALL-E and other compatible image generation models.

Activity Name: litellm_image_generation

Use Cases: Creative image generation, illustrations, marketing materials

litellm_function_call​

Function calling with tool-compatible models for structured interactions.

Activity Name: litellm_function_call

Use Cases: API integrations, structured data extraction, workflow automation

litellm_batch​

Process multiple requests efficiently in batches with automatic error handling.

Activity Name: litellm_batch

Use Cases: Bulk processing, data analysis at scale, parallel operations

Supported Providers​

Major Cloud Providers​

  • OpenAI: GPT-4, GPT-3.5, DALL-E, Embeddings
  • Anthropic: Claude 3.5 Sonnet, Claude 3 Haiku, Claude 3 Opus
  • Google: Gemini Pro, Gemini Flash, PaLM
  • Azure OpenAI: Enterprise GPT models with custom deployments
  • AWS Bedrock: Claude, Titan, Jurassic models

Local & Open Source​

  • Ollama: Local model hosting (Llama, Mistral, CodeLlama)
  • Hugging Face: Transformers and Inference API
  • Together AI: Open source models at scale
  • Replicate: Community and custom models

Specialized Providers​

  • Cohere: Command models and embeddings
  • AI21: Jurassic models for text generation
  • Anyscale: Ray-powered model serving
  • Perplexity: Search-enhanced language models

Configuration​

Authentication​

LiteLLM uses provider-specific authentication:

{
  "api_key_secret": "OPENAI_API_KEY",
  "api_base": "https://api.openai.com/v1",
  "organization": "org-id"
}

Authentication Patterns:

  • Provider Keys: OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY
  • Unified Key: LITELLM_API_KEY for proxy deployments
  • Secrets Manager: Recommended for production environments

Model Naming​

LiteLLM uses consistent model naming across providers:

{
  "model": "gpt-4"
}

Provider-Specific Examples:

  • OpenAI: gpt-4, gpt-3.5-turbo, dall-e-3
  • Anthropic: claude-3-5-sonnet-20241022, claude-3-haiku-20240307
  • Google: gemini-pro, gemini-pro-vision
  • Local: ollama/llama2, together_ai/llama-2-7b

OpenAI Configuration​

Configure OpenAI models with standard authentication:

{
  "model": "gpt-4",
  "api_key_secret": "OPENAI_API_KEY",
  "organization": "org-your-organization-id",
  "api_base": "https://api.openai.com/v1"
}

Required Settings:

  • api_key_secret: Reference to OpenAI API key in secrets manager
  • model: OpenAI model identifier (e.g., gpt-4, gpt-3.5-turbo)

Optional Settings:

  • organization: OpenAI organization ID for usage tracking
  • api_base: Custom API endpoint (for Azure OpenAI or proxies)
  • api_version: API version for Azure OpenAI deployments

Anthropic Configuration​

Configure Anthropic Claude models with secure authentication:

{
  "model": "claude-3-5-sonnet-20241022",
  "api_key_secret": "ANTHROPIC_API_KEY",
  "max_tokens": 1000
}

Required Settings:

  • api_key_secret: Reference to Anthropic API key in secrets manager
  • model: Claude model identifier (e.g., claude-3-5-sonnet-20241022)

Best Practices:

  • Use latest model versions for optimal performance
  • Set appropriate max_tokens limits for cost control
  • Store API keys securely using secrets management

Step Documentation​

litellm_chat​

Universal chat completions with consistent interface across all providers.

Configuration​

{
  "activity": "litellm_chat",
  "model": "gpt-4",
  "messages": [
    {"role": "user", "content": "Hello!"}
  ]
}

Parameters​

  • model (string, default: gpt-3.5-turbo) - Model identifier
  • messages (array) - Chat messages in OpenAI format
  • prompt (string) - Alternative to messages for simple prompts
  • temperature (float, default: 1.0) - Randomness control
  • max_tokens (int) - Maximum response length
  • stream (boolean, default: false) - Enable streaming responses

Input Patterns​

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Explain quantum computing"}
  ]
}

Example​

{
  "name": "generate_response",
  "activity": "litellm_chat",
  "config": {
    "model": "claude-3-5-sonnet-20241022",
    "messages": [
      {"role": "user", "content": "Write a product description for wireless earbuds"}
    ],
    "temperature": 0.7,
    "max_tokens": 500
  }
}

litellm_vision​

Image analysis using vision-capable models from multiple providers.

Configuration​

{
  "activity": "litellm_vision",
  "model": "gpt-4-vision-preview",
  "image_path": "previous_step.outputs.images[0].path",
  "prompt": "Describe this image"
}

Parameters​

  • model (string, default: gpt-4-vision-preview) - Vision model
  • image_path (string, required) - Path to image file
  • prompt (string, default: "Analyze this image") - Analysis instruction
  • detail (string, default: "auto") - Image detail level for processing
  • max_tokens (int, default: 300) - Maximum response length

Supported Models​

  • OpenAI: gpt-4-vision-preview, gpt-4o
  • Google: gemini-pro-vision
  • Anthropic: claude-3-opus, claude-3-sonnet (with vision)

Example​

{
  "name": "analyze_screenshot",
  "activity": "litellm_vision",
  "config": {
    "model": "gpt-4-vision-preview",
    "image_path": "screenshot.outputs.image_path",
    "prompt": "Identify any UI issues or accessibility problems in this interface",
    "detail": "high"
  }
}

litellm_embeddings​

Generate embeddings using various providers and models.

Configuration​

{
  "activity": "litellm_embeddings",
  "model": "text-embedding-ada-002",
  "input": "Text to embed"
}

Parameters​

  • model (string, default: text-embedding-ada-002) - Embedding model
  • input (string, required) - Text to convert to embeddings
  • encoding_format (string) - Encoding format for embeddings
  • dimensions (int) - Dimension size for supported models
  • OpenAI: text-embedding-ada-002, text-embedding-3-small, text-embedding-3-large
  • Cohere: embed-english-v3.0, embed-multilingual-v3.0
  • Google: textembedding-gecko@001

Example​

{
  "name": "create_embeddings",
  "activity": "litellm_embeddings",
  "config": {
    "model": "text-embedding-3-small",
    "input_path": "document_reader.outputs.text",
    "dimensions": 1536
  }
}

litellm_image_generation​

Generate images using DALL-E and compatible models.

Configuration​

{
  "activity": "litellm_image_generation",
  "model": "dall-e-3",
  "prompt": "A futuristic cityscape at sunset"
}

Parameters​

  • model (string, default: dall-e-3) - Image generation model
  • prompt (string, required) - Image description
  • n (int, default: 1) - Number of images to generate
  • size (string, default: 1024x1024) - Image dimensions
  • quality (string) - Image quality setting
  • style (string) - Art style preference

Supported Models​

  • OpenAI: dall-e-3, dall-e-2
  • Together AI: Various Stable Diffusion models
  • Replicate: Community image generation models

Example​

{
  "name": "create_marketing_image",
  "activity": "litellm_image_generation",
  "config": {
    "model": "dall-e-3",
    "prompt": "Professional product photo of wireless earbuds on a clean white background",
    "size": "1792x1024",
    "quality": "hd"
  }
}

litellm_function_call​

Structured interactions using function calling capabilities.

Configuration​

{
  "activity": "litellm_function_call",
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "What's the weather?"}],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get weather information",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type": "string"}
          }
        }
      }
    }
  ]
}

Parameters​

  • model (string, default: gpt-4) - Function-capable model
  • messages (array, required) - Conversation messages
  • tools (array, required) - Available function definitions
  • tool_choice (string, default: auto) - Function selection strategy
  • parallel_tool_calls (boolean) - Allow multiple simultaneous calls

Example​

{
  "name": "api_interaction",
  "activity": "litellm_function_call",
  "config": {
    "model": "gpt-4",
    "messages": [
      {"role": "user", "content": "Send an email to john@example.com about the meeting"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "send_email",
          "description": "Send an email message",
          "parameters": {
            "type": "object",
            "properties": {
              "to": {"type": "string"},
              "subject": {"type": "string"},
              "body": {"type": "string"}
            },
            "required": ["to", "subject", "body"]
          }
        }
      }
    ]
  }
}

litellm_batch​

Process multiple requests efficiently with automatic batching and error handling.

Configuration​

{
  "activity": "litellm_batch",
  "model": "gpt-3.5-turbo",
  "requests": [
    {"messages": [{"role": "user", "content": "Prompt 1"}]},
    {"messages": [{"role": "user", "content": "Prompt 2"}]}
  ],
  "batch_size": 5
}

Parameters​

  • model (string, default: gpt-3.5-turbo) - Model for all requests
  • requests (array, required) - Array of request configurations
  • batch_size (int, default: 5) - Maximum concurrent requests
  • requests_path (string) - Alternative path to requests array

Example​

{
  "name": "process_feedback",
  "activity": "litellm_batch",
  "config": {
    "model": "gpt-3.5-turbo",
    "requests_path": "survey_data.outputs.responses",
    "batch_size": 10
  }
}

Advanced Usage​

Provider Switching​

{
  "steps": [
    {
      "name": "fast_generation",
      "activity": "litellm_chat",
      "config": {
        "model": "gpt-3.5-turbo",
        "prompt": "Generate initial ideas"
      }
    },
    {
      "name": "quality_refinement", 
      "activity": "litellm_chat",
      "config": {
        "model": "claude-3-5-sonnet-20241022",
        "prompt": "Refine these ideas: {{fast_generation.outputs.text}}"
      }
    }
  ]
}

Multi-Modal Workflow​

{
  "steps": [
    {
      "name": "generate_image",
      "activity": "litellm_image_generation",
      "config": {
        "model": "dall-e-3",
        "prompt": "Product mockup design"
      }
    },
    {
      "name": "analyze_image",
      "activity": "litellm_vision",
      "config": {
        "model": "gpt-4-vision-preview",
        "image_path": "generate_image.outputs.images[0].path",
        "prompt": "Provide design feedback and suggestions"
      }
    }
  ]
}

Error Handling​

Common Issues​

  • Authentication Error: Verify provider-specific API keys
  • Model Not Available: Check model name and provider support
  • Rate Limiting: Automatic retry with exponential backoff
  • Token Limits: Respect model-specific context limits

Best Practices​

  • Use appropriate models for task complexity and cost requirements
  • Implement fallback providers for critical workflows
  • Monitor usage and costs across different providers
  • Test with multiple providers to optimize performance

Performance Tips​

Model Selection​

  • Speed Priority: gpt-3.5-turbo, claude-3-haiku, gemini-flash
  • Quality Priority: gpt-4, claude-3-5-sonnet, gemini-pro
  • Cost Optimization: Compare pricing across providers for your use case

Batch Processing​

  • Use litellm_batch for processing multiple similar requests
  • Adjust batch_size based on provider rate limits
  • Implement proper error handling for partial failures

Integration Examples​

View complete workflow examples in the Flow Library:

  • Multi-provider AI pipelines
  • Cost-optimized workflows
  • Function calling integrations
  • Batch processing examples