Shared Endpoint Settings

This page describes the shared configuration settings for all endpoints. The settings highlighted here are available to all configurations under the “Endpoints” field unless noted otherwise.

Example Configuration

endpoints:
  # Individual endpoint configurations
  openAI:
    streamRate: 25
    titleModel: "gpt-4o-mini"
    titleMethod: "completion"
    titlePrompt: "Create a concise title for this conversation:\n\n{convo}"
  
  azureOpenAI:
    streamRate: 35
    titleModel: "grok-3"
    titleMethod: "structured"
    titlePrompt: |
      Analyze this conversation and provide:
      1. A concise title in the detected language (5 words or less, no punctuation or quotation)
      2. Always provide a relevant emoji at the start of the title
 
      {convo}
    titleConvo: true
  
  anthropic:
    streamRate: 25
    titleModel: "claude-3-5-haiku-20241022"
    titleMethod: "completion"
  
  bedrock:
    streamRate: 25
    titleModel: "us.amazon.nova-lite-v1:0"
    titleEndpoint: "anthropic"
  
  google:
    streamRate: 1
    titleModel: "gemini-2.0-flash-lite"
    titlePromptTemplate: "Human: {input}\nAssistant: {output}"
  
  assistants:
    streamRate: 30
  
  azureAssistants:
    streamRate: 30
  
  # Global configuration using 'all' - this will override ALL individual endpoint settings that are "shared",
  # i.e. streamRate, titleModel, titleMethod, titlePrompt, titlePromptTemplate, titleEndpoint.
  all:
    titleConvo: true
    titleModel: "gpt-4.1-nano"
    titlePrompt: |
      Analyze this conversation and provide:
      1. The detected language of the conversation
      2. A concise title in the detected language (5 words or less, no punctuation or quotation)
      3. Always provide a relevant emoji at the start of the title

Important: When using the all configuration, it will override ALL individual endpoint settings for the properties you define. In the example above, the all configuration would override the titleConvo, titleModel, and titlePrompt settings for all endpoints, while individual streamRate settings would be preserved since it’s not defined in all.

streamRate

Key:

Default: 1

Allows for streaming data at the fastest rate possible while allowing the system to wait for the next tick

titleConvo

Key:

Default: false

Notes:

• When enabled, titles will be generated automatically using the configured title settings
• Must be used in conjunction with titleModel or the endpoint must have a default model available

Example:

titleConvo: true

titleModel

Key:

Default: System default for the current endpoint

titleMethod

Key:

Default: "completion"

Available Methods:

• "completion" - Uses standard completion API without tools/functions. Compatible with most LLMs.
• "structured" - Uses structured output for title generation. Requires provider/model support.
• "functions" - Legacy alias for “structured”. Functionally identical.

Example:

titleMethod: "completion"

titlePrompt

Key:

Default:

Analyze this conversation and provide:
1. The detected language of the conversation
2. A concise title in the detected language (5 words or less, no punctuation or quotation)

{convo}

Notes:

• Must always include the {convo} placeholder
• The {convo} placeholder will be replaced with the formatted conversation
• Can be placed anywhere in the prompt

Example:

titlePrompt: "Create a brief, descriptive title for the following conversation:\n\n{convo}\n\nTitle:"

titlePromptTemplate

Key:

Default: "User: {input}\nAI: {output}"

Notes:

• Must include both {input} and {output} placeholders
• {input} is replaced with the user’s initial message
• {output} is replaced with the AI’s response
• The formatted result replaces {convo} in the titlePrompt

Example:

titlePromptTemplate: "Human: {input}\n\nAssistant: {output}"

titleEndpoint

Key:

Default: Uses the current conversation’s endpoint

Accepted Values:

• openAI
• azureOpenAI
• google
• anthropic
• bedrock
• For custom endpoints: use the exact custom endpoint name

Example:

# Use Anthropic for titles even when chatting with OpenAI
endpoints:
  openAI:
    titleEndpoint: "anthropic"
    # Will use anthropic's configuration for title generation

Notes:

• All settings shown on this page can be configured individually per endpoint or globally using the all key
• When using the all configuration, it will override the corresponding settings in ALL individual endpoints
• Settings not defined in all will preserve their individual endpoint values
• For streamRate: Recommended values are between 25-40 for a smooth streaming experience
• Using a higher stream rate is a must when serving the app to many users at scale

Example of Override Behavior:

endpoints:
  openAI:
    streamRate: 25      # This will be preserved
    titleModel: "gpt-4" # This will be overridden
    titleConvo: false   # This will be overridden
  
  all:
    titleConvo: true
    titleModel: "gpt-3.5-turbo"
    # streamRate not defined here, so individual values are kept

Endpoint Settings

• Custom Endpoints
• OpenAI
• Anthropic
• Bedrock
• Google
• Azure OpenAI
• Assistants